[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:09:59 dist-test-slave-95hh krb5kdc[16881](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:09:59 dist-test-slave-95hh krb5kdc[16881](info): set up 2 sockets
Sep 23 23:09:59 dist-test-slave-95hh krb5kdc[16881](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260923 23:10:00.419947 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.948s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:00.530272 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42591
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:42591
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:00.609875 16894 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:00.610117 16894 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:00.610152 16894 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:00.611348 16894 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:00.611394 16894 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:00.611413 16894 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:00.611428 16894 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:00.611443 16894 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:00.612950 16894 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:42591
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42591
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16894
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:00.613217 16894 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:00.613440 16894 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:00.618134 16894 server_base.cc:1034] running on GCE node
W20260923 23:10:00.618077 16902 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.618068 16899 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.618068 16900 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:00.618573 16894 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:00.618882 16894 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:00.620035 16894 hybrid_clock.cc:648] HybridClock initialized: now 1790205000620000 us; error 47 us; skew 500 ppm
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:00.622411 16894 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:00.622875 16894 webserver.cc:460] Webserver started at http://127.16.122.190:44013/ using document root <none> and password file <none>
I20260923 23:10:00.623137 16894 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:00.623190 16894 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:00.623319 16894 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:00.624347 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "9197c5222a8b4cebabc5a57c4a9f4973"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "bc2f1ca1a2cd67d1b146d7b978c18f9b"
server_key_iv: "b9ff9b5ae4cd86bf548ebd049cd8d928"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.624691 16894 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "9197c5222a8b4cebabc5a57c4a9f4973"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "bc2f1ca1a2cd67d1b146d7b978c18f9b"
server_key_iv: "b9ff9b5ae4cd86bf548ebd049cd8d928"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.625885 16894 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:00.626513 16909 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.626650 16894 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:00.626731 16894 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "9197c5222a8b4cebabc5a57c4a9f4973"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "bc2f1ca1a2cd67d1b146d7b978c18f9b"
server_key_iv: "b9ff9b5ae4cd86bf548ebd049cd8d928"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.626806 16894 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:00.654938 16894 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:00.656764 16894 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:00.656901 16894 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:00.660645 16894 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:42591
I20260923 23:10:00.660681 16961 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:42591 every 8 connection(s)
I20260923 23:10:00.661006 16894 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:00.661824 16962 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:00.664074 16962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973: Bootstrap starting.
I20260923 23:10:00.664431 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 16894
I20260923 23:10:00.664515 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:00.664661 16962 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:00.664911 16962 log.cc:824] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:00.665156 16874 external_mini_cluster.cc:1428] Setting key 9605368b88e74dfb9b6cfd9352eba5b1
I20260923 23:10:00.665637 16962 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973: No bootstrap required, opened a new log
I20260923 23:10:00.667868 16962 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } }
I20260923 23:10:00.668043 16962 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:00.668084 16962 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9197c5222a8b4cebabc5a57c4a9f4973, State: Initialized, Role: FOLLOWER
I20260923 23:10:00.668210 16962 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } }
I20260923 23:10:00.668286 16962 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:00.668321 16962 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:00.668366 16962 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:00.669358 16962 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } }
I20260923 23:10:00.669464 16962 leader_election.cc:302] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9197c5222a8b4cebabc5a57c4a9f4973; no voters:
I20260923 23:10:00.669615 16962 leader_election.cc:288] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:00.669646 16967 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:00.669785 16967 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [term 1 LEADER]: Becoming Leader. State: Replica: 9197c5222a8b4cebabc5a57c4a9f4973, State: Running, Role: LEADER
I20260923 23:10:00.669857 16962 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:00.669868 16967 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } }
I20260923 23:10:00.670243 16968 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } } }
I20260923 23:10:00.670320 16968 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:00.670470 16969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9197c5222a8b4cebabc5a57c4a9f4973. Latest consensus state: current_term: 1 leader_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9197c5222a8b4cebabc5a57c4a9f4973" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42591 } } }
I20260923 23:10:00.670528 16969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:00.670636 16974 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:00.671157 16974 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:00.673892 16974 catalog_manager.cc:1347] Generated new cluster ID: ff98fd12091a45a8a3b9b0040f652a1d
I20260923 23:10:00.673938 16974 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:00.681990 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:00.666024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36270 (local address 127.16.122.190:42591)
0923 23:10:00.666197 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:00.666205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:00.666582 (+ 377us) server_negotiation.cc:407] Connection header received
0923 23:10:00.668844 (+ 2262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:00.668850 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:00.668890 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:00.669004 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:00.671203 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.672542 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.674002 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.674405 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.674499 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:00.677850 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:00.677855 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:00.677855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:00.677867 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:00.679887 (+ 2020us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.680423 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.680426 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.680426 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.680556 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.680861 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.680862 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.680862 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.681162 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:00.681314 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:00.681630 (+ 316us) server_negotiation.cc:299] Negotiation successful
0923 23:10:00.681776 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":88,"threads_started":1}
I20260923 23:10:00.682593 16974 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:00.683234 16974 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:00.686579 16974 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9197c5222a8b4cebabc5a57c4a9f4973: Generated new TSK 0
I20260923 23:10:00.686743 16974 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:00.710182 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:00.789091 16988 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:00.789320 16988 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:00.789351 16988 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:00.790499 16988 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:00.790540 16988 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:00.790578 16988 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:00.792052 16988 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.16988
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:00.792301 16988 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:00.792512 16988 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:00.798115 16996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.798115 16993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.798141 16994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:00.798345 16988 server_base.cc:1034] running on GCE node
I20260923 23:10:00.798560 16988 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:00.798760 16988 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:00.799901 16988 hybrid_clock.cc:648] HybridClock initialized: now 1790205000799888 us; error 29 us; skew 500 ppm
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:00.802210 16988 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:00.802649 16988 webserver.cc:460] Webserver started at http://127.16.122.129:40271/ using document root <none> and password file <none>
I20260923 23:10:00.802856 16988 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:00.802903 16988 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:00.803048 16988 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:00.804076 16988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "66d34af6d90643b6a607a65f5e9dd417"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "d88342a6d1b6e4585d5ab110b929afa4"
server_key_iv: "12dbefc8f1239ec6f942660672bc8cf1"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.804412 16988 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "66d34af6d90643b6a607a65f5e9dd417"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "d88342a6d1b6e4585d5ab110b929afa4"
server_key_iv: "12dbefc8f1239ec6f942660672bc8cf1"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.805619 16988 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:00.806246 17003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.806394 16988 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:00.806474 16988 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "66d34af6d90643b6a607a65f5e9dd417"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "d88342a6d1b6e4585d5ab110b929afa4"
server_key_iv: "12dbefc8f1239ec6f942660672bc8cf1"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.806535 16988 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:00.812865 16988 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:00.814527 16988 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:00.814663 16988 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:00.814865 16988 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:00.815188 16988 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:00.815222 16988 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.815251 16988 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:00.815274 16988 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.820749 16988 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:39425
I20260923 23:10:00.820797 17116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:39425 every 8 connection(s)
I20260923 23:10:00.821099 16988 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:00.825011 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 16988
I20260923 23:10:00.825091 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:00.825243 16874 external_mini_cluster.cc:1428] Setting key f2a9688cfb9cce7277709b3a9303858e
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:00.830364 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:00.821758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:47373 (local address 127.16.122.190:42591)
0923 23:10:00.821807 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:00.821809 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:00.821982 (+ 173us) server_negotiation.cc:407] Connection header received
0923 23:10:00.822506 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:00.822507 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:00.822524 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:00.822575 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:00.823578 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.824116 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.825115 (+ 999us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.825383 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.825440 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:00.827731 (+ 2291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:00.827733 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:00.827733 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:00.827740 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:00.829075 (+ 1335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.829442 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.829442 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.829443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.829519 (+ 76us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.829721 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.829721 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.829722 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.829852 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:00.829922 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:00.830188 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 23:10:00.830270 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:00.830449 17119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:00.821729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42591 (local address 127.16.122.129:47373)
0923 23:10:00.821866 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:00.821903 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:00.822405 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:00.822649 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:00.822653 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:00.823279 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:00.823479 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:00.823484 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.824206 (+ 722us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.824207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:00.825029 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:00.825032 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.825453 (+ 421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.825454 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:00.825589 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:00.826112 (+ 523us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:00.826120 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:00.827643 (+ 1523us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:00.829152 (+ 1509us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:00.829159 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:00.829161 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:00.829381 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:00.829574 (+ 193us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:00.829575 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:00.829575 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:00.829668 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:00.829909 (+ 241us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:00.829912 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:00.830066 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 23:10:00.830148 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 23:10:00.830275 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":67,"threads_started":1}
I20260923 23:10:00.830926 17117 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42591
I20260923 23:10:00.831013 17117 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:00.831209 17117 heartbeater.cc:511] Master 127.16.122.190:42591 requested a full tablet report, sending...
I20260923 23:10:00.831756 16926 ts_manager.cc:194] Registered new tserver with Master: 66d34af6d90643b6a607a65f5e9dd417 (127.16.122.129:39425)
I20260923 23:10:00.832671 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:47373
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:00.850450 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:00.929911 17122 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:00.930114 17122 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:00.930135 17122 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:00.931285 17122 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:00.931327 17122 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:00.931353 17122 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:00.932804 17122 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17122
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:00.932996 17122 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:00.933182 17122 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:00.937803 17130 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.937800 17127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:00.937831 17128 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:00.937852 17122 server_base.cc:1034] running on GCE node
I20260923 23:10:00.938175 17122 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:00.938377 17122 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:00.939504 17122 hybrid_clock.cc:648] HybridClock initialized: now 1790205000939495 us; error 27 us; skew 500 ppm
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:00.941731 17122 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:00.942152 17122 webserver.cc:460] Webserver started at http://127.16.122.130:38679/ using document root <none> and password file <none>
I20260923 23:10:00.942345 17122 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:00.942386 17122 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:00.942504 17122 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:00.943535 17122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "789f24a7850a418eaff8df968032d051"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "0b1c4b93b933de56d73abf2c92268ab8"
server_key_iv: "3be4a9533dc6976d0389f7837d599f2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.943845 17122 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "789f24a7850a418eaff8df968032d051"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "0b1c4b93b933de56d73abf2c92268ab8"
server_key_iv: "3be4a9533dc6976d0389f7837d599f2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.945065 17122 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:00.945701 17137 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.945866 17122 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:00.945945 17122 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "789f24a7850a418eaff8df968032d051"
format_stamp: "Formatted at 2026-09-23 23:10:00 on dist-test-slave-95hh"
server_key: "0b1c4b93b933de56d73abf2c92268ab8"
server_key_iv: "3be4a9533dc6976d0389f7837d599f2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:00.945994 17122 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:00.959375 17122 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:00.961241 17122 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:00.961362 17122 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:00.961556 17122 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:00.961825 17122 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:00.961864 17122 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.961889 17122 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:00.961907 17122 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:00.967777 17122 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:40389
I20260923 23:10:00.967834 17250 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:40389 every 8 connection(s)
I20260923 23:10:00.968134 17122 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:00 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205000, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:00.974931 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17122
I20260923 23:10:00.975013 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:00.975207 16874 external_mini_cluster.cc:1428] Setting key 213661b99319f47cfd109506b80ca092
I20260923 23:10:00.977005 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:00.968842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56827 (local address 127.16.122.190:42591)
0923 23:10:00.968894 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:00.968896 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:00.969091 (+ 195us) server_negotiation.cc:407] Connection header received
0923 23:10:00.969576 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:00.969578 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:00.969589 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:00.969636 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:00.970593 (+ 957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.971190 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.972176 (+ 986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.972356 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.972412 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:00.974389 (+ 1977us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:00.974390 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:00.974391 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:00.974398 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:00.975696 (+ 1298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.976155 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.976156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.976156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.976206 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:00.976448 (+ 242us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:00.976449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:00.976449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:00.976548 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:00.976592 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:00.976881 (+ 289us) server_negotiation.cc:299] Negotiation successful
0923 23:10:00.976941 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:10:00.977254 17253 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:00.968808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42591 (local address 127.16.122.130:56827)
0923 23:10:00.968979 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:00.969014 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:00.969485 (+ 471us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:00.969716 (+ 231us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:00.969721 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:00.970314 (+ 593us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:00.970501 (+ 187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:00.970506 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.971276 (+ 770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.971277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:00.972092 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:00.972095 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:00.972423 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:00.972424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:00.972551 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:00.973005 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:00.973013 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:00.974302 (+ 1289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:00.975775 (+ 1473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:00.975782 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:00.975784 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:00.976077 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:00.976261 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:00.976262 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:00.976262 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:00.976399 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:00.976595 (+ 196us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:00.976599 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:00.976765 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 23:10:00.976869 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 23:10:00.977025 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260923 23:10:00.977679 17251 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42591
I20260923 23:10:00.977784 17251 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:00.977965 17251 heartbeater.cc:511] Master 127.16.122.190:42591 requested a full tablet report, sending...
I20260923 23:10:00.978322 16926 ts_manager.cc:194] Registered new tserver with Master: 789f24a7850a418eaff8df968032d051 (127.16.122.130:40389)
I20260923 23:10:00.978772 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:56827
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:00.998215 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34149
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:01.077006 17256 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:01.077219 17256 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:01.077241 17256 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:01.078323 17256 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:01.078357 17256 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:01.078382 17256 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:01.079823 17256 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34149
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42591
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17256
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:01.080017 17256 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:01.080200 17256 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:01.084987 17256 server_base.cc:1034] running on GCE node
W20260923 23:10:01.084946 17262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:01.084950 17264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:01.084947 17261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:01.085316 17256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:01.085558 17256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:01.086691 17256 hybrid_clock.cc:648] HybridClock initialized: now 1790205001086675 us; error 34 us; skew 500 ppm
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:01.088990 17256 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:01.089355 17256 webserver.cc:460] Webserver started at http://127.16.122.131:39491/ using document root <none> and password file <none>
I20260923 23:10:01.089535 17256 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:01.089573 17256 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:01.089663 17256 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:01.090636 17256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "227a1b41b01b4db4b176b55fc5c0d87e"
format_stamp: "Formatted at 2026-09-23 23:10:01 on dist-test-slave-95hh"
server_key: "002998b80ab91d722a5670cb86639a2e"
server_key_iv: "be10f9b10b6d8d30f5b27fe3cbebc053"
server_key_version: "encryptionkey@0"
I20260923 23:10:01.090938 17256 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "227a1b41b01b4db4b176b55fc5c0d87e"
format_stamp: "Formatted at 2026-09-23 23:10:01 on dist-test-slave-95hh"
server_key: "002998b80ab91d722a5670cb86639a2e"
server_key_iv: "be10f9b10b6d8d30f5b27fe3cbebc053"
server_key_version: "encryptionkey@0"
I20260923 23:10:01.092156 17256 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:01.092805 17271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:01.092936 17256 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:01.093003 17256 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "227a1b41b01b4db4b176b55fc5c0d87e"
format_stamp: "Formatted at 2026-09-23 23:10:01 on dist-test-slave-95hh"
server_key: "002998b80ab91d722a5670cb86639a2e"
server_key_iv: "be10f9b10b6d8d30f5b27fe3cbebc053"
server_key_version: "encryptionkey@0"
I20260923 23:10:01.093055 17256 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:01.116775 17256 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:01.118921 17256 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:01.119069 17256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:01.119266 17256 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:01.119545 17256 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:01.119577 17256 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:01.119607 17256 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:01.119630 17256 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:01.124966 17256 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:41269
I20260923 23:10:01.125010 17384 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:41269 every 8 connection(s)
I20260923 23:10:01.125334 17256 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:01.132565 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17256
I20260923 23:10:01.132647 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:01.132791 16874 external_mini_cluster.cc:1428] Setting key 2a03b29220933758007c5ae1ac49b004
I20260923 23:10:01.133996 16965 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:01.126040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:38939 (local address 127.16.122.190:42591)
0923 23:10:01.126093 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:01.126095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:01.126292 (+ 197us) server_negotiation.cc:407] Connection header received
0923 23:10:01.126764 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:01.126765 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:01.126776 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:01.126837 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:01.127777 (+ 940us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.128257 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.129198 (+ 941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.129385 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.129428 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:01.131409 (+ 1981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:01.131410 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:01.131411 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:01.131417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:01.132648 (+ 1231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.133068 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.133069 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.133069 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.133120 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.133386 (+ 266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.133387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.133387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.133521 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:01.133581 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:01.133862 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 23:10:01.133933 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:10:01.134222 17387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:01.126015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42591 (local address 127.16.122.131:38939)
0923 23:10:01.126172 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:01.126209 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:01.126686 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:01.126896 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:01.126900 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:01.127498 (+ 598us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:01.127705 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:01.127710 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.128321 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.128321 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:01.129128 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:01.129131 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.129439 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.129440 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:01.129546 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:01.129990 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:01.129998 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:01.131330 (+ 1332us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:01.132721 (+ 1391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:01.132729 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:01.132731 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:01.132997 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:01.133178 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:01.133179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:01.133179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:01.133327 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:01.133577 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:01.133581 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:01.133735 (+ 154us) client_negotiation.cc:769] Sending connection context
0923 23:10:01.133828 (+ 93us) client_negotiation.cc:240] Negotiation successful
0923 23:10:01.133993 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":90,"threads_started":1}
I20260923 23:10:01.134604 17385 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42591
I20260923 23:10:01.134684 17385 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:01.134848 17385 heartbeater.cc:511] Master 127.16.122.190:42591 requested a full tablet report, sending...
I20260923 23:10:01.135185 16926 ts_manager.cc:194] Registered new tserver with Master: 227a1b41b01b4db4b176b55fc5c0d87e (127.16.122.131:41269)
I20260923 23:10:01.135535 16926 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:38939
I20260923 23:10:01.144749 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:01.165803 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:01.158925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35868 (local address 127.16.122.129:39425)
0923 23:10:01.159124 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:01.159129 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:01.159140 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:01.159166 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:01.159168 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:01.159183 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:01.159256 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:01.159742 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.160486 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.161150 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.161448 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.161500 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:01.163515 (+ 2015us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:01.163523 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:01.163524 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:01.163537 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:01.164890 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.165172 (+ 282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.165174 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.165174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.165239 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.165366 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.165367 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.165367 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.165489 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:01.165601 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:01.165635 (+ 34us) server_negotiation.cc:299] Negotiation successful
0923 23:10:01.165699 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":94,"threads_started":1}
W20260923 23:10:01.166146 17031 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35868
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[16881](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:01.186431 17392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:01.180203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35880 (local address 127.16.122.129:39425)
0923 23:10:01.180256 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:01.180258 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:01.180324 (+ 66us) server_negotiation.cc:407] Connection header received
0923 23:10:01.180397 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:01.180399 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:01.180409 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:01.180448 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:01.180930 (+ 482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.181399 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.182066 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:01.182336 (+ 270us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:01.182393 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:01.184425 (+ 2032us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:01.184426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:01.184426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:01.184436 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:01.185500 (+ 1064us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.185779 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.185780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.185781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.185831 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:01.185985 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:01.185986 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:01.185986 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:01.186082 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:01.186184 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:01.186258 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:10:01.186338 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:10:01.186955 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 16988
I20260923 23:10:01.190757 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17122
I20260923 23:10:01.194618 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17256
I20260923 23:10:01.198067 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 16894
2026-09-23T23:10:01Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (1750 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[17399](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[17399](info): set up 2 sockets
Sep 23 23:10:01 dist-test-slave-95hh krb5kdc[17399](info): commencing operation
krb5kdc: starting...
W20260923 23:10:02.155239 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:02.262140 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34713
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34713
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:02.341378 17412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:02.341598 17412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:02.341619 17412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:02.342792 17412 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:02.342840 17412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:02.342854 17412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:02.342877 17412 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:02.342898 17412 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:02.344429 17412 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34713
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34713
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17412
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:02.344687 17412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:02.344907 17412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:02.349712 17418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.349789 17412 server_base.cc:1034] running on GCE node
W20260923 23:10:02.349723 17420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:02.349727 17417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.350126 17412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:02.350405 17412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:02.351536 17412 hybrid_clock.cc:648] HybridClock initialized: now 1790205002351527 us; error 31 us; skew 500 ppm
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:02.353811 17412 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:02.354174 17412 webserver.cc:460] Webserver started at http://127.16.122.190:40739/ using document root <none> and password file <none>
I20260923 23:10:02.354363 17412 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:02.354405 17412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:02.354521 17412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:02.355585 17412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "e9074ba207464e8284ca0242107e0f49"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "ac5d16fd393e5645a077fabf9c75fddf"
server_key_iv: "84cd238d929de0dda59d4690e2351fc7"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.355909 17412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "e9074ba207464e8284ca0242107e0f49"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "ac5d16fd393e5645a077fabf9c75fddf"
server_key_iv: "84cd238d929de0dda59d4690e2351fc7"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.357107 17412 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:02.357789 17427 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.357939 17412 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:02.358013 17412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "e9074ba207464e8284ca0242107e0f49"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "ac5d16fd393e5645a077fabf9c75fddf"
server_key_iv: "84cd238d929de0dda59d4690e2351fc7"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.358078 17412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:02.385320 17412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:02.387168 17412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:02.387296 17412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:02.390950 17412 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34713
I20260923 23:10:02.390996 17479 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34713 every 8 connection(s)
I20260923 23:10:02.391371 17412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:02.392212 17480 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:02.394474 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49: Bootstrap starting.
I20260923 23:10:02.395018 17480 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:02.395306 17480 log.cc:824] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:02.395977 17480 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49: No bootstrap required, opened a new log
I20260923 23:10:02.396209 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17412
I20260923 23:10:02.396276 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:02.396411 16874 external_mini_cluster.cc:1428] Setting key 86773cd713147c6f8a5dd095b65fd7f5
I20260923 23:10:02.397989 17480 raft_consensus.cc:348] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } }
I20260923 23:10:02.398170 17480 raft_consensus.cc:374] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:02.398208 17480 raft_consensus.cc:729] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9074ba207464e8284ca0242107e0f49, State: Initialized, Role: FOLLOWER
I20260923 23:10:02.398334 17480 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } }
I20260923 23:10:02.398404 17480 raft_consensus.cc:388] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:02.398427 17480 raft_consensus.cc:482] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:02.398469 17480 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:02.399569 17480 raft_consensus.cc:504] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } }
I20260923 23:10:02.399678 17480 leader_election.cc:302] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e9074ba207464e8284ca0242107e0f49; no voters:
I20260923 23:10:02.399844 17480 leader_election.cc:288] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:02.399881 17485 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:02.400012 17485 raft_consensus.cc:686] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [term 1 LEADER]: Becoming Leader. State: Replica: e9074ba207464e8284ca0242107e0f49, State: Running, Role: LEADER
I20260923 23:10:02.400070 17480 sys_catalog.cc:564] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:02.400101 17485 consensus_queue.cc:237] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } }
I20260923 23:10:02.400545 17487 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e9074ba207464e8284ca0242107e0f49. Latest consensus state: current_term: 1 leader_uuid: "e9074ba207464e8284ca0242107e0f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } } }
I20260923 23:10:02.400625 17487 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:02.400758 17486 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e9074ba207464e8284ca0242107e0f49" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e9074ba207464e8284ca0242107e0f49" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34713 } } }
I20260923 23:10:02.400812 17486 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:02.400923 17494 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:02.401584 17494 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:02.403119 17494 catalog_manager.cc:1347] Generated new cluster ID: b333e39612f54e71aeb7e9a17b06bbb3
I20260923 23:10:02.403168 17494 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:02.407424 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.396924 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52208 (local address 127.16.122.190:34713)
0923 23:10:02.397114 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.397124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.397162 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:10:02.397953 (+ 791us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.397962 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.398028 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.398182 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:02.398847 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.400241 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.401333 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.401958 (+ 625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.402053 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.404419 (+ 2366us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:02.404423 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:02.404423 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:02.404435 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:02.405906 (+ 1471us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.406222 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.406225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.406225 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.406323 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.406490 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.406491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.406491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.406726 (+ 235us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:02.406875 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.406969 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.407127 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":82,"threads_started":1}
I20260923 23:10:02.420555 17494 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:02.421046 17494 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:02.427584 17494 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e9074ba207464e8284ca0242107e0f49: Generated new TSK 0
I20260923 23:10:02.427763 17494 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:02.459080 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:02.555176 17506 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:02.555371 17506 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:02.555401 17506 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:02.556535 17506 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:02.556576 17506 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:02.556609 17506 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:02.558050 17506 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17506
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:02.558290 17506 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:02.558493 17506 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:02.563436 17512 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.563474 17506 server_base.cc:1034] running on GCE node
W20260923 23:10:02.563436 17514 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:02.563458 17511 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.563858 17506 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:02.564083 17506 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:02.565218 17506 hybrid_clock.cc:648] HybridClock initialized: now 1790205002565203 us; error 35 us; skew 500 ppm
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:02.567595 17506 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:02.567986 17506 webserver.cc:460] Webserver started at http://127.16.122.129:43583/ using document root <none> and password file <none>
I20260923 23:10:02.568189 17506 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:02.568238 17506 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:02.568367 17506 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:02.569386 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "406052f9b6a1467483c0b1497321933c"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "a4aa4c40b01f0a44ec69e706a3a936c3"
server_key_iv: "af569a21c80fc1fe95583fbfc4af8b83"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.569695 17506 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "406052f9b6a1467483c0b1497321933c"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "a4aa4c40b01f0a44ec69e706a3a936c3"
server_key_iv: "af569a21c80fc1fe95583fbfc4af8b83"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.570922 17506 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:02.571660 17521 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.571832 17506 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.571910 17506 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "406052f9b6a1467483c0b1497321933c"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "a4aa4c40b01f0a44ec69e706a3a936c3"
server_key_iv: "af569a21c80fc1fe95583fbfc4af8b83"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.571971 17506 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:02.584044 17506 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:02.585708 17506 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:02.585845 17506 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:02.586041 17506 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:02.586315 17506 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:02.586349 17506 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.586376 17506 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:02.586393 17506 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.591722 17506 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:40133
I20260923 23:10:02.591777 17634 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:40133 every 8 connection(s)
I20260923 23:10:02.592070 17506 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:02.594132 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17506
I20260923 23:10:02.594228 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:02.594372 16874 external_mini_cluster.cc:1428] Setting key 8e80666a9a35206ec643cd2c89831ce9
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:10:02.601639 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.592744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:45313 (local address 127.16.122.190:34713)
0923 23:10:02.592805 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.592807 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.592984 (+ 177us) server_negotiation.cc:407] Connection header received
0923 23:10:02.593877 (+ 893us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.593878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.593889 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.593945 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:02.594724 (+ 779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.595318 (+ 594us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.596617 (+ 1299us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.596917 (+ 300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.596983 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.598970 (+ 1987us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:02.598971 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:02.598971 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:02.598981 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:02.600314 (+ 1333us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.600704 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.600705 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.600706 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.600759 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.601042 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.601043 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.601043 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.601158 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:02.601212 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.601464 (+ 252us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.601548 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:10:02.601830 17637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.592709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34713 (local address 127.16.122.129:45313)
0923 23:10:02.592860 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.592901 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.593788 (+ 887us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.594024 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.594029 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.594388 (+ 359us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:02.594648 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.594655 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.595403 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.595404 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.596536 (+ 1132us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.596538 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.596983 (+ 445us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.596984 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.597084 (+ 100us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.597559 (+ 475us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:02.597567 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:02.598885 (+ 1318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:02.600385 (+ 1500us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.600391 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.600393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.600610 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.600813 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.600814 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.600814 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.600988 (+ 174us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.601211 (+ 223us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:02.601218 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:02.601351 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.601439 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.601603 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":81,"threads_started":1}
I20260923 23:10:02.602233 17635 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34713
I20260923 23:10:02.602320 17635 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:02.602459 17635 heartbeater.cc:511] Master 127.16.122.190:34713 requested a full tablet report, sending...
I20260923 23:10:02.602916 17444 ts_manager.cc:194] Registered new tserver with Master: 406052f9b6a1467483c0b1497321933c (127.16.122.129:40133)
I20260923 23:10:02.603504 17444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:45313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:02.617440 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:02.694479 17640 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:02.694681 17640 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:02.694712 17640 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:02.695848 17640 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:02.695886 17640 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:02.695920 17640 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:02.697326 17640 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17640
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:02.697548 17640 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:02.697757 17640 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:02.702378 17648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.702513 17640 server_base.cc:1034] running on GCE node
W20260923 23:10:02.702344 17645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:02.702344 17646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.702817 17640 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:02.703068 17640 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:02.704208 17640 hybrid_clock.cc:648] HybridClock initialized: now 1790205002704196 us; error 30 us; skew 500 ppm
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:02.706501 17640 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:02.706897 17640 webserver.cc:460] Webserver started at http://127.16.122.130:34261/ using document root <none> and password file <none>
I20260923 23:10:02.707119 17640 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:02.707168 17640 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:02.707295 17640 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:02.708303 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "482b7ac62421440b8c456c32da7f2bc9"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "38bc548b4278930db2c97c08d9e0ac8f"
server_key_iv: "d596140a4f26e0183f361bdb361a917a"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.708621 17640 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "482b7ac62421440b8c456c32da7f2bc9"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "38bc548b4278930db2c97c08d9e0ac8f"
server_key_iv: "d596140a4f26e0183f361bdb361a917a"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.709813 17640 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:02.710398 17655 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.710537 17640 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:02.710611 17640 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "482b7ac62421440b8c456c32da7f2bc9"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "38bc548b4278930db2c97c08d9e0ac8f"
server_key_iv: "d596140a4f26e0183f361bdb361a917a"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.710668 17640 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:02.719415 17640 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:02.720954 17640 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:02.721073 17640 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:02.721264 17640 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:02.721536 17640 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:02.721568 17640 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.721594 17640 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:02.721611 17640 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.726665 17640 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:36207
I20260923 23:10:02.726722 17768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:36207 every 8 connection(s)
I20260923 23:10:02.726995 17640 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:02.731439 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17640
I20260923 23:10:02.731525 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:02.731686 16874 external_mini_cluster.cc:1428] Setting key 12967ea16852b92798e35622f3ca86a5
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:02.735575 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.727612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:49275 (local address 127.16.122.190:34713)
0923 23:10:02.727677 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.727679 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.727851 (+ 172us) server_negotiation.cc:407] Connection header received
0923 23:10:02.728678 (+ 827us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.728679 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.728687 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.728735 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:02.729273 (+ 538us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.729753 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.730621 (+ 868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.730799 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.730839 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.732997 (+ 2158us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:02.732998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:02.732999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:02.733005 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:02.734425 (+ 1420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.734756 (+ 331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.734757 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.734757 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.734813 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.734993 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.734993 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.734994 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.735129 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:02.735190 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.735405 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.735485 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:10:02.735679 17771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.727588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34713 (local address 127.16.122.130:49275)
0923 23:10:02.727725 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.727764 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.728596 (+ 832us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.728803 (+ 207us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.728806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.729034 (+ 228us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:02.729214 (+ 180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.729218 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.729810 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.729811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.730565 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.730567 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.730849 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.730850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.730940 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.731379 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:02.731389 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:02.732916 (+ 1527us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:02.734496 (+ 1580us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.734501 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.734503 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.734694 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.734865 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.734866 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.734866 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.734947 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.735181 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:02.735184 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:02.735295 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.735373 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.735483 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":61,"threads_started":1}
I20260923 23:10:02.736093 17769 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34713
I20260923 23:10:02.736171 17769 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:02.736331 17769 heartbeater.cc:511] Master 127.16.122.190:34713 requested a full tablet report, sending...
I20260923 23:10:02.736665 17444 ts_manager.cc:194] Registered new tserver with Master: 482b7ac62421440b8c456c32da7f2bc9 (127.16.122.130:36207)
I20260923 23:10:02.737133 17444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:49275
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:02.754782 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33353
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:02.832850 17774 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:02.833055 17774 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:02.833084 17774 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:02.834236 17774 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:02.834280 17774 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:02.834307 17774 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:02.835793 17774 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33353
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34713
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17774
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:02.836020 17774 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:02.836249 17774 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:02.840881 17780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:02.840871 17779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.841064 17774 server_base.cc:1034] running on GCE node
W20260923 23:10:02.840871 17782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:02.841341 17774 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:02.841559 17774 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:02.842695 17774 hybrid_clock.cc:648] HybridClock initialized: now 1790205002842684 us; error 27 us; skew 500 ppm
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:02.844939 17774 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:02.845331 17774 webserver.cc:460] Webserver started at http://127.16.122.131:33105/ using document root <none> and password file <none>
I20260923 23:10:02.845530 17774 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:02.845574 17774 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:02.845695 17774 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:02.846669 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "3505057317fc4e44b0e6cb5e790806ff"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "99c5d1d7b848530b42ad6ff210831605"
server_key_iv: "2eb0a2c66d556f8991854a9c93bbe482"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.846997 17774 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "3505057317fc4e44b0e6cb5e790806ff"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "99c5d1d7b848530b42ad6ff210831605"
server_key_iv: "2eb0a2c66d556f8991854a9c93bbe482"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.848206 17774 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:02.848891 17789 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.849025 17774 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:02.849097 17774 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "3505057317fc4e44b0e6cb5e790806ff"
format_stamp: "Formatted at 2026-09-23 23:10:02 on dist-test-slave-95hh"
server_key: "99c5d1d7b848530b42ad6ff210831605"
server_key_iv: "2eb0a2c66d556f8991854a9c93bbe482"
server_key_version: "encryptionkey@0"
I20260923 23:10:02.849159 17774 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:02.858083 17774 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:02.859723 17774 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:02.859840 17774 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:02.860018 17774 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:02.860292 17774 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:02.860330 17774 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.860350 17774 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:02.860373 17774 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:02.865653 17774 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:35507
I20260923 23:10:02.865712 17902 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:35507 every 8 connection(s)
I20260923 23:10:02.865959 17774 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:02.868731 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17774
I20260923 23:10:02.868815 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:02.868960 16874 external_mini_cluster.cc:1428] Setting key b3effbfd92627921688745d83aa93c2f
Sep 23 23:10:02 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:02.874284 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.866579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:49241 (local address 127.16.122.190:34713)
0923 23:10:02.866629 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.866631 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.866800 (+ 169us) server_negotiation.cc:407] Connection header received
0923 23:10:02.867756 (+ 956us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.867757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.867770 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.867827 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:02.868412 (+ 585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.868956 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.870139 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.870335 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.870385 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.872286 (+ 1901us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:02.872287 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:02.872288 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:02.872294 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:02.873320 (+ 1026us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.873632 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.873633 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.873633 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.873675 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.873829 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.873829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.873829 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.873917 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:02.873959 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.874153 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.874221 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:10:02.874513 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.866560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34713 (local address 127.16.122.131:49241)
0923 23:10:02.866694 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.866727 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.867646 (+ 919us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.867897 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.867901 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.868142 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:02.868324 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.868328 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.869056 (+ 728us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.869058 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.870070 (+ 1012us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.870072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.870394 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.870395 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.870486 (+ 91us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.870874 (+ 388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:02.870881 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:02.872204 (+ 1323us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:02.873380 (+ 1176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.873386 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.873387 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.873580 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.873710 (+ 130us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:02.873711 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:02.873711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:02.873791 (+ 80us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:02.873958 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:02.873960 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:02.874063 (+ 103us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.874136 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.874244 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":58,"threads_started":1}
I20260923 23:10:02.874884 17903 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34713
I20260923 23:10:02.874966 17903 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:02.875154 17903 heartbeater.cc:511] Master 127.16.122.190:34713 requested a full tablet report, sending...
I20260923 23:10:02.875504 17444 ts_manager.cc:194] Registered new tserver with Master: 3505057317fc4e44b0e6cb5e790806ff (127.16.122.131:35507)
I20260923 23:10:02.875890 17444 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:49241
I20260923 23:10:02.880405 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:10:02.886757 17484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.881826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52220 (local address 127.16.122.190:34713)
0923 23:10:02.881874 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.881875 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.881991 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:10:02.882030 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.882031 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.882040 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.882081 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:02.882572 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.883062 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.883714 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.883890 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.883936 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.884885 (+ 949us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:02.884886 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:02.884886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:02.884892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:02.885926 (+ 1034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.886203 (+ 277us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.886204 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.886204 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.886253 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:02.886365 (+ 112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:02.886366 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:02.886366 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:02.886490 (+ 124us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:02.886557 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.886625 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.886682 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:02.888760 17444 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52220:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:02.890223 17444 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:02.895987 17913 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892210 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40368 (local address 127.16.122.131:35507)
0923 23:10:02.892383 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.892388 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.892758 (+ 370us) server_negotiation.cc:407] Connection header received
0923 23:10:02.892807 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.892809 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.892824 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.892887 (+ 63us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:02.893059 (+ 172us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.893768 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895147 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.895638 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895702 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.895844 (+ 142us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.895880 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.895914 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":81,"threads_started":1}
I20260923 23:10:02.896008 17911 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:35507 (local address 127.0.0.1:40368)
0923 23:10:02.892522 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.892531 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.892583 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.892965 (+ 382us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.892966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.892968 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:02.892995 (+ 27us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.892996 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.893865 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.893866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.895063 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.895065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.895709 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895710 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.895789 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.895803 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.895855 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.895906 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":381,"mutex_wait_us":2,"thread_start_us":67,"threads_started":1}
I20260923 23:10:02.896145 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44328 (local address 127.16.122.130:36207)
0923 23:10:02.892712 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.892717 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.892743 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:10:02.892776 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.892778 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.892792 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.892864 (+ 72us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:02.892991 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.894014 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.894942 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.895650 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895715 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.895765 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.896046 (+ 281us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.896080 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":154,"thread_start_us":83,"threads_started":1}
I20260923 23:10:02.896178 17912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36207 (local address 127.0.0.1:44328)
0923 23:10:02.892669 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.892676 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.892712 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.892862 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.892863 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.892865 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:02.892916 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.892918 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.894126 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.894127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.894853 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.894855 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.895798 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895799 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.895857 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.895861 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.895907 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.895946 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":73,"threads_started":1}
I20260923 23:10:02.896431 17915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:40133 (local address 127.0.0.1:34456)
0923 23:10:02.892454 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.892486 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.892564 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.892753 (+ 189us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.892756 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.892759 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:02.892821 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.892822 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.894110 (+ 1288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.894111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.895176 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.895178 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.896229 (+ 1051us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.896230 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.896287 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.896291 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.896340 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.896373 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":298,"thread_start_us":71,"threads_started":1}
I20260923 23:10:02.896472 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.892226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34456 (local address 127.16.122.129:40133)
0923 23:10:02.892468 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.892472 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.892563 (+ 91us) server_negotiation.cc:407] Connection header received
0923 23:10:02.892648 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.892651 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.892667 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.892744 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:02.892901 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.894015 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895241 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.895925 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.895980 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.896025 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.896366 (+ 341us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.896403 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":208,"thread_start_us":69,"threads_started":1}
I20260923 23:10:02.896737 17837 tablet_service.cc:1467] Processing CreateTablet for tablet 2e644e0d389d4b2789ce478d220c35b2 (DEFAULT_TABLE table=test-table [id=824fc289840c481996b89f442e98038b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:02.896732 17703 tablet_service.cc:1467] Processing CreateTablet for tablet 2e644e0d389d4b2789ce478d220c35b2 (DEFAULT_TABLE table=test-table [id=824fc289840c481996b89f442e98038b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:02.896963 17569 tablet_service.cc:1467] Processing CreateTablet for tablet 2e644e0d389d4b2789ce478d220c35b2 (DEFAULT_TABLE table=test-table [id=824fc289840c481996b89f442e98038b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:02.897060 17703 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e644e0d389d4b2789ce478d220c35b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:02.897060 17837 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e644e0d389d4b2789ce478d220c35b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:02.897228 17569 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2e644e0d389d4b2789ce478d220c35b2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:02.900164 17918 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Bootstrap starting.
I20260923 23:10:02.900550 17919 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: Bootstrap starting.
I20260923 23:10:02.900796 17918 tablet_bootstrap.cc:654] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:02.901080 17919 tablet_bootstrap.cc:654] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:02.901271 17918 log.cc:824] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:02.901578 17919 log.cc:824] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:02.901857 17917 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: Bootstrap starting.
I20260923 23:10:02.901985 17918 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: No bootstrap required, opened a new log
I20260923 23:10:02.902052 17918 ts_tablet_manager.cc:1397] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:02.902369 17917 tablet_bootstrap.cc:654] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:02.902411 17919 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: No bootstrap required, opened a new log
I20260923 23:10:02.902473 17919 ts_tablet_manager.cc:1397] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:02.902801 17917 log.cc:824] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:02.903419 17917 tablet_bootstrap.cc:492] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: No bootstrap required, opened a new log
I20260923 23:10:02.903475 17917 ts_tablet_manager.cc:1397] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:02.903967 17918 raft_consensus.cc:348] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.904148 17918 raft_consensus.cc:374] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:02.904183 17918 raft_consensus.cc:729] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 482b7ac62421440b8c456c32da7f2bc9, State: Initialized, Role: FOLLOWER
I20260923 23:10:02.904317 17918 consensus_queue.cc:260] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.904593 17918 ts_tablet_manager.cc:1428] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:10:02.904728 17769 heartbeater.cc:503] Master 127.16.122.190:34713 was elected leader, sending a full tablet report...
I20260923 23:10:02.904699 17919 raft_consensus.cc:348] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.904840 17919 raft_consensus.cc:374] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:02.904862 17919 raft_consensus.cc:729] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3505057317fc4e44b0e6cb5e790806ff, State: Initialized, Role: FOLLOWER
I20260923 23:10:02.904944 17919 consensus_queue.cc:260] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.905023 17917 raft_consensus.cc:348] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.905176 17903 heartbeater.cc:503] Master 127.16.122.190:34713 was elected leader, sending a full tablet report...
I20260923 23:10:02.905185 17917 raft_consensus.cc:374] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:02.905222 17917 raft_consensus.cc:729] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 406052f9b6a1467483c0b1497321933c, State: Initialized, Role: FOLLOWER
I20260923 23:10:02.905335 17917 consensus_queue.cc:260] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.905534 17919 ts_tablet_manager.cc:1428] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:10:02.905597 17917 ts_tablet_manager.cc:1428] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:02.905753 17635 heartbeater.cc:503] Master 127.16.122.190:34713 was elected leader, sending a full tablet report...
W20260923 23:10:02.977490 17770 tablet.cc:2369] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:02.979662 17924 raft_consensus.cc:482] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:02.979760 17924 raft_consensus.cc:504] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.980113 17924 leader_election.cc:288] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 406052f9b6a1467483c0b1497321933c (127.16.122.129:40133), 482b7ac62421440b8c456c32da7f2bc9 (127.16.122.130:36207)
I20260923 23:10:02.982792 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.980364 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:57673 (local address 127.16.122.129:40133)
0923 23:10:02.980416 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.980418 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.980428 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:02.980523 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.980525 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.980535 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.980596 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:02.980728 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.981198 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982134 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.982487 (+ 353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982548 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.982556 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.982683 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.982719 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:02.982788 17905 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.980306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:40133 (local address 127.16.122.131:57673)
0923 23:10:02.980364 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.980375 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.980418 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.980608 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.980609 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.980611 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:02.980673 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.980675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.981263 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.981264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.982049 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.982052 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.982554 (+ 502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982555 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.982621 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.982623 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.982669 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.982710 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:10:02.983013 17916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.980385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:52503 (local address 127.16.122.130:36207)
0923 23:10:02.980424 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:02.980426 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:02.980590 (+ 164us) server_negotiation.cc:407] Connection header received
0923 23:10:02.980650 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:02.980651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:02.980659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:02.980699 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:02.980831 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.981280 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982201 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.982558 (+ 357us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982620 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.982628 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:02.982781 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 23:10:02.982822 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:02.983230 17926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:02.980313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36207 (local address 127.16.122.131:52503)
0923 23:10:02.980524 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:02.980533 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:02.980592 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:02.980710 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:02.980711 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:02.980713 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:02.980775 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.980776 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.981352 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.981353 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.982119 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:02.982121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:02.982631 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:02.982632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:02.982698 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:02.982701 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:02.983117 (+ 416us) client_negotiation.cc:240] Negotiation successful
0923 23:10:02.983158 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":12,"thread_start_us":128,"threads_started":1}
I20260923 23:10:02.983179 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e644e0d389d4b2789ce478d220c35b2" candidate_uuid: "3505057317fc4e44b0e6cb5e790806ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406052f9b6a1467483c0b1497321933c" is_pre_election: true
I20260923 23:10:02.983348 17589 raft_consensus.cc:2454] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3505057317fc4e44b0e6cb5e790806ff in term 0.
I20260923 23:10:02.983551 17723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e644e0d389d4b2789ce478d220c35b2" candidate_uuid: "3505057317fc4e44b0e6cb5e790806ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b7ac62421440b8c456c32da7f2bc9" is_pre_election: true
I20260923 23:10:02.983582 17790 leader_election.cc:302] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3505057317fc4e44b0e6cb5e790806ff, 406052f9b6a1467483c0b1497321933c; no voters:
I20260923 23:10:02.983668 17723 raft_consensus.cc:2454] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 3505057317fc4e44b0e6cb5e790806ff in term 0.
I20260923 23:10:02.983709 17924 raft_consensus.cc:2781] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:02.983770 17924 raft_consensus.cc:482] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:02.983798 17924 raft_consensus.cc:3037] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:02.984453 17924 raft_consensus.cc:504] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.984558 17924 leader_election.cc:288] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [CANDIDATE]: Term 1 election: Requested vote from peers 406052f9b6a1467483c0b1497321933c (127.16.122.129:40133), 482b7ac62421440b8c456c32da7f2bc9 (127.16.122.130:36207)
I20260923 23:10:02.984695 17589 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e644e0d389d4b2789ce478d220c35b2" candidate_uuid: "3505057317fc4e44b0e6cb5e790806ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "406052f9b6a1467483c0b1497321933c"
I20260923 23:10:02.984714 17723 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2e644e0d389d4b2789ce478d220c35b2" candidate_uuid: "3505057317fc4e44b0e6cb5e790806ff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "482b7ac62421440b8c456c32da7f2bc9"
I20260923 23:10:02.984772 17589 raft_consensus.cc:3037] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:02.984783 17723 raft_consensus.cc:3037] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:02.985411 17589 raft_consensus.cc:2454] T 2e644e0d389d4b2789ce478d220c35b2 P 406052f9b6a1467483c0b1497321933c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3505057317fc4e44b0e6cb5e790806ff in term 1.
I20260923 23:10:02.985486 17723 raft_consensus.cc:2454] T 2e644e0d389d4b2789ce478d220c35b2 P 482b7ac62421440b8c456c32da7f2bc9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 3505057317fc4e44b0e6cb5e790806ff in term 1.
I20260923 23:10:02.985594 17790 leader_election.cc:302] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3505057317fc4e44b0e6cb5e790806ff, 406052f9b6a1467483c0b1497321933c; no voters:
I20260923 23:10:02.985690 17924 raft_consensus.cc:2781] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:02.985842 17924 raft_consensus.cc:686] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [term 1 LEADER]: Becoming Leader. State: Replica: 3505057317fc4e44b0e6cb5e790806ff, State: Running, Role: LEADER
I20260923 23:10:02.985942 17924 consensus_queue.cc:237] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } }
I20260923 23:10:02.986743 17444 catalog_manager.cc:5495] T 2e644e0d389d4b2789ce478d220c35b2 P 3505057317fc4e44b0e6cb5e790806ff reported cstate change: term changed from 0 to 1, leader changed from <none> to 3505057317fc4e44b0e6cb5e790806ff (127.16.122.131). New cstate: current_term: 1 leader_uuid: "3505057317fc4e44b0e6cb5e790806ff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3505057317fc4e44b0e6cb5e790806ff" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35507 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "406052f9b6a1467483c0b1497321933c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40133 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "482b7ac62421440b8c456c32da7f2bc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36207 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205002, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:03.013705 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:03.007351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34472 (local address 127.16.122.129:40133)
0923 23:10:03.007422 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:03.007423 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:03.007488 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:10:03.007546 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:03.007547 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:03.007555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:03.007592 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:03.008103 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.008564 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.009195 (+ 631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.009374 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.009415 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:03.011424 (+ 2009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:03.011434 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:03.011434 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:03.011440 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:03.012826 (+ 1386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.013093 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.013096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.013096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.013157 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.013294 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.013294 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.013295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.013418 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:03.013510 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:03.013558 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 23:10:03.013621 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205003, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:03.034060 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:03.027930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34476 (local address 127.16.122.129:40133)
0923 23:10:03.027988 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:03.027990 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:03.028052 (+ 62us) server_negotiation.cc:407] Connection header received
0923 23:10:03.028129 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:03.028130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:03.028138 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:03.028202 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:03.028694 (+ 492us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.029156 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.029845 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.030027 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.030080 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:03.032027 (+ 1947us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:03.032028 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:03.032028 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:03.032036 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:03.033073 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.033356 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.033357 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.033357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.033405 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.033559 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.033561 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.033561 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.033649 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:03.033731 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:03.033892 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 23:10:03.033967 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17399](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17399](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205003, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:03.054010 17914 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:03.047811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34480 (local address 127.16.122.129:40133)
0923 23:10:03.047865 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:03.047867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:03.047953 (+ 86us) server_negotiation.cc:407] Connection header received
0923 23:10:03.048016 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:03.048017 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:03.048027 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:03.048084 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:03.048609 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.049085 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.049781 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:03.049967 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:03.050020 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:03.052017 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:03.052018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:03.052018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:03.052025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:03.053191 (+ 1166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.053490 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.053491 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.053491 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.053535 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:03.053657 (+ 122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:03.053657 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:03.053658 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:03.053746 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:03.053810 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:03.053889 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 23:10:03.053941 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260923 23:10:03.054474 17549 tablet_service.cc:2908] Rejecting scan request for tablet 2e644e0d389d4b2789ce478d220c35b2: Uninitialized: clean time has not yet been initialized
I20260923 23:10:03.055298 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17506
I20260923 23:10:03.060070 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17640
I20260923 23:10:03.064126 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17774
I20260923 23:10:03.068043 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17412
2026-09-23T23:10:03Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (1870 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17941](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17941](info): set up 2 sockets
Sep 23 23:10:03 dist-test-slave-95hh krb5kdc[17941](info): commencing operation
krb5kdc: starting...
W20260923 23:10:04.018925 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:04.126552 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35595
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:35595
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:04.204509 17954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:04.204721 17954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:04.204751 17954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:04.206063 17954 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:04.206115 17954 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:10:04.206130 17954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:04.206146 17954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:04.206167 17954 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:04.206192 17954 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:04.207762 17954 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:35595
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35595
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.17954
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:04.208010 17954 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:04.208230 17954 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:04.212693 17962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.212733 17960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.212743 17959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.212893 17954 server_base.cc:1034] running on GCE node
I20260923 23:10:04.213142 17954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:04.213385 17954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:04.214526 17954 hybrid_clock.cc:648] HybridClock initialized: now 1790205004214508 us; error 29 us; skew 500 ppm
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:04.216928 17954 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:04.217334 17954 webserver.cc:460] Webserver started at http://127.16.122.190:33801/ using document root <none> and password file <none>
I20260923 23:10:04.217507 17954 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:04.217543 17954 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:04.217633 17954 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:04.218590 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "fd2e09af0b1d447ba23bdc8020e3b727"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "b58c73bc4e8e68023ae2e0d690e46669"
server_key_iv: "ce4af4e9a993365eddcb2cc09fc1ba23"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.218870 17954 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "fd2e09af0b1d447ba23bdc8020e3b727"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "b58c73bc4e8e68023ae2e0d690e46669"
server_key_iv: "ce4af4e9a993365eddcb2cc09fc1ba23"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.220041 17954 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:04.220669 17969 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.220811 17954 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:04.220890 17954 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "fd2e09af0b1d447ba23bdc8020e3b727"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "b58c73bc4e8e68023ae2e0d690e46669"
server_key_iv: "ce4af4e9a993365eddcb2cc09fc1ba23"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.220943 17954 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:04.236431 17954 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:04.238556 17954 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:04.238670 17954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:04.242779 17954 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:35595
I20260923 23:10:04.242856 18031 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:35595 every 8 connection(s)
I20260923 23:10:04.243166 17954 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:04.243799 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.246001 18032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Bootstrap starting.
I20260923 23:10:04.246557 18032 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.246814 18032 log.cc:824] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:04.247499 18032 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: No bootstrap required, opened a new log
I20260923 23:10:04.249526 18032 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } }
I20260923 23:10:04.249662 18032 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.249702 18032 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd2e09af0b1d447ba23bdc8020e3b727, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.249841 18032 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } }
I20260923 23:10:04.249920 18032 raft_consensus.cc:388] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:04.249953 18032 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:04.249998 18032 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:04.250496 18032 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } }
I20260923 23:10:04.250588 18032 leader_election.cc:302] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fd2e09af0b1d447ba23bdc8020e3b727; no voters:
I20260923 23:10:04.250712 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 17954
I20260923 23:10:04.250756 18032 leader_election.cc:288] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:04.250797 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:04.250895 18035 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:04.250942 16874 external_mini_cluster.cc:1428] Setting key 9fa6599664a4422810c8cafcbace4c43
I20260923 23:10:04.250998 18032 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:04.251052 18035 raft_consensus.cc:686] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [term 1 LEADER]: Becoming Leader. State: Replica: fd2e09af0b1d447ba23bdc8020e3b727, State: Running, Role: LEADER
I20260923 23:10:04.251149 18035 consensus_queue.cc:237] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } }
I20260923 23:10:04.251470 18035 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fd2e09af0b1d447ba23bdc8020e3b727. Latest consensus state: current_term: 1 leader_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } } }
I20260923 23:10:04.251538 18035 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:04.251652 18037 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fd2e09af0b1d447ba23bdc8020e3b727" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35595 } } }
I20260923 23:10:04.251818 18037 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:04.251732 18046 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:04.252414 18046 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:04.254984 18046 catalog_manager.cc:1347] Generated new cluster ID: e0c5a9ea8b1a4ee1ac411d5653859ca8
I20260923 23:10:04.255059 18046 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:04.260387 18046 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:04.261039 18046 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:04.261266 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.251424 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41560 (local address 127.16.122.190:35595)
0923 23:10:04.251969 (+ 545us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.251978 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.252009 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:10:04.252127 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.252135 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.252194 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.253068 (+ 874us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.253173 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.254484 (+ 1311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.255309 (+ 825us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.255724 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.255819 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.257927 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.257931 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.257931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.257942 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.259761 (+ 1819us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.260100 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.260104 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.260104 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.260216 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.260424 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.260425 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.260426 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.260669 (+ 243us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.260818 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.260899 (+ 81us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.261030 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":66,"threads_started":1}
I20260923 23:10:04.271517 18046 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Generated new TSK 0
I20260923 23:10:04.271714 18046 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:04.302059 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35595
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:04.406658 18058 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:04.406879 18058 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:04.406903 18058 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:04.408093 18058 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:04.408133 18058 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:04.408154 18058 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:04.408177 18058 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:04.409582 18058 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:35595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18058
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:04.409806 18058 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:04.410001 18058 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:04.414630 18063 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.414696 18058 server_base.cc:1034] running on GCE node
W20260923 23:10:04.414638 18064 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.414639 18066 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.415077 18058 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:04.415329 18058 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:04.416513 18058 hybrid_clock.cc:648] HybridClock initialized: now 1790205004416497 us; error 27 us; skew 500 ppm
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:04.418864 18058 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:04.419302 18058 webserver.cc:460] Webserver started at http://127.16.122.129:44111/ using document root <none> and password file <none>
I20260923 23:10:04.419514 18058 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:04.419561 18058 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:04.419682 18058 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:04.420686 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "2b40d4071f61406187f038f775009ad1"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "fc324f60bab1c2df46a8fcf13119131b"
server_key_iv: "e5476e3b123659f78cd555ebf309d31f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.421020 18058 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "2b40d4071f61406187f038f775009ad1"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "fc324f60bab1c2df46a8fcf13119131b"
server_key_iv: "e5476e3b123659f78cd555ebf309d31f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.422164 18058 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:04.422765 18073 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.422896 18058 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:04.422974 18058 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "2b40d4071f61406187f038f775009ad1"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "fc324f60bab1c2df46a8fcf13119131b"
server_key_iv: "e5476e3b123659f78cd555ebf309d31f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.423060 18058 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:04.432219 18058 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:04.433811 18058 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:04.433941 18058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:04.434394 18058 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:04.434435 18058 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.434463 18058 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:04.434479 18058 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.440594 18058 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42329
I20260923 23:10:04.440666 18188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42329 every 8 connection(s)
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:04.440940 18058 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:04.443527 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.434828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:41035 (local address 127.16.122.190:35595)
0923 23:10:04.434868 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.434870 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.435104 (+ 234us) server_negotiation.cc:407] Connection header received
0923 23:10:04.435599 (+ 495us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.435600 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.435618 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.435687 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.436456 (+ 769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.437199 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.438624 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.438916 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.438989 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.441381 (+ 2392us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.441383 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.441383 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.441394 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.442446 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.442784 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.442785 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.442785 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.442832 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.443002 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.443002 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.443003 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.443136 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.443185 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.443401 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.443463 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:04.443697 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.434833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.129:41035)
0923 23:10:04.434970 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.435004 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.435505 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.435776 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.435779 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.436089 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.436356 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.436362 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.437287 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.437288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.438536 (+ 1248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.438539 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.438988 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.438989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.439134 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.439623 (+ 489us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.439633 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.441289 (+ 1656us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.442512 (+ 1223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.442517 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.442519 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.442727 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.442874 (+ 147us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.442875 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.442875 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.442961 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.443187 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.443191 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.443303 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.443383 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.443499 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":79,"threads_started":1}
I20260923 23:10:04.444135 18189 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35595
I20260923 23:10:04.444213 18189 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:04.444366 18189 heartbeater.cc:511] Master 127.16.122.190:35595 requested a full tablet report, sending...
I20260923 23:10:04.444862 17985 ts_manager.cc:194] Registered new tserver with Master: 2b40d4071f61406187f038f775009ad1 (127.16.122.129:42329)
I20260923 23:10:04.445516 17985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:41035
I20260923 23:10:04.447180 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18058
I20260923 23:10:04.447270 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:04.447436 16874 external_mini_cluster.cc:1428] Setting key d618654a909be8f56c82d6db1b333931
I20260923 23:10:04.450212 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.445056 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:58741 (local address 127.16.122.190:35595)
0923 23:10:04.445093 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.445095 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.445206 (+ 111us) server_negotiation.cc:407] Connection header received
0923 23:10:04.445259 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.445260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.445269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.445309 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.445588 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.446095 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.446747 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.446948 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.447023 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.447946 (+ 923us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.447947 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.447948 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.447953 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.449227 (+ 1274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.449557 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.449558 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.449558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.449613 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.449765 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.449765 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.449766 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.449873 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.449927 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.450055 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.450123 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 23:10:04.450218 18195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.445022 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.129:58741)
0923 23:10:04.445142 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.445152 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.445206 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.445321 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.445322 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.445484 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.445530 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.445531 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.446177 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.446178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.446673 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.446674 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.447021 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.447022 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.447120 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.447550 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.447555 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.447877 (+ 322us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.449294 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.449295 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.449302 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.449483 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.449660 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.449661 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.449661 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.449717 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.449924 (+ 207us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.449925 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.449970 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.450028 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.450123 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":72,"threads_started":1}
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:04.469884 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35595
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:04.549661 18198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:04.549877 18198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:04.549901 18198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:04.551122 18198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:04.551164 18198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:04.551183 18198 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:04.551199 18198 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:04.552661 18198 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:35595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18198
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:04.552884 18198 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:04.553087 18198 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:04.557857 18204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.557942 18198 server_base.cc:1034] running on GCE node
W20260923 23:10:04.557951 18206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.557862 18203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.558240 18198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:04.558470 18198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:04.559594 18198 hybrid_clock.cc:648] HybridClock initialized: now 1790205004559586 us; error 27 us; skew 500 ppm
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:04.565218 18198 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:04.565657 18198 webserver.cc:460] Webserver started at http://127.16.122.130:39459/ using document root <none> and password file <none>
I20260923 23:10:04.565866 18198 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:04.565915 18198 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:04.566042 18198 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:04.567169 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "72ea34b9211547ae8ebe7ba9e59894b5"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "487a3991bf684b84f1161295cc232246"
server_key_iv: "783c76d4cfa69636ee7ac9ae6a2afc9f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.567507 18198 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "72ea34b9211547ae8ebe7ba9e59894b5"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "487a3991bf684b84f1161295cc232246"
server_key_iv: "783c76d4cfa69636ee7ac9ae6a2afc9f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.568754 18198 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:04.569420 18213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.569562 18198 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:04.569648 18198 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "72ea34b9211547ae8ebe7ba9e59894b5"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "487a3991bf684b84f1161295cc232246"
server_key_iv: "783c76d4cfa69636ee7ac9ae6a2afc9f"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.569718 18198 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:04.588112 18198 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:04.589865 18198 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:04.590008 18198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:04.590505 18198 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:04.590545 18198 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.590574 18198 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:04.590591 18198 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:04.597026 18198 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:42765
I20260923 23:10:04.597420 18198 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:04.597405 18328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:42765 every 8 connection(s)
I20260923 23:10:04.599340 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.590926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:49737 (local address 127.16.122.190:35595)
0923 23:10:04.590972 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.590974 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.591217 (+ 243us) server_negotiation.cc:407] Connection header received
0923 23:10:04.591708 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.591709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.591722 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.591773 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.592561 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.593323 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.594340 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.594634 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.594703 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.596834 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.596835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.596835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.596843 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.598279 (+ 1436us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.598602 (+ 323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.598602 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.598603 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.598650 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.598810 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.598811 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.598811 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.598907 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.598950 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.599216 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.599273 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:04.599499 18222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.590937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.130:49737)
0923 23:10:04.591104 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.591142 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.591617 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.591854 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.591858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.592173 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.592445 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.592452 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.593418 (+ 966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.593419 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.594245 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.594247 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.594721 (+ 474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.594722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.594861 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.595415 (+ 554us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.595422 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.596742 (+ 1320us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.598339 (+ 1597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.598342 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.598344 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.598542 (+ 198us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.598686 (+ 144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.598686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.598686 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.598768 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.598949 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.598952 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.599102 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.599192 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.599315 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":96,"threads_started":1}
I20260923 23:10:04.599911 18329 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35595
I20260923 23:10:04.599987 18329 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:04.600147 18329 heartbeater.cc:511] Master 127.16.122.190:35595 requested a full tablet report, sending...
I20260923 23:10:04.600476 17986 ts_manager.cc:194] Registered new tserver with Master: 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765)
I20260923 23:10:04.600914 17986 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:49737
I20260923 23:10:04.604584 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18198
I20260923 23:10:04.604662 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:04.604837 16874 external_mini_cluster.cc:1428] Setting key 625013bb954261aedb3c38bfe609086c
I20260923 23:10:04.605820 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.600876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:52919 (local address 127.16.122.190:35595)
0923 23:10:04.600951 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.600952 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.601008 (+ 56us) server_negotiation.cc:407] Connection header received
0923 23:10:04.601064 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.601065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.601074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.601124 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.601390 (+ 266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.601844 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.602482 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.602675 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.602725 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.603536 (+ 811us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.603537 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.603537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.603543 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.604789 (+ 1246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.605095 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.605096 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.605096 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.605149 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.605309 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.605310 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.605310 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.605474 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.605531 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.605665 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.605731 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 23:10:04.605837 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.600825 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.130:52919)
0923 23:10:04.600952 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.600962 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.601020 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.601127 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.601128 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.601274 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.601326 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.601327 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.601921 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.601922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.602417 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.602419 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.602727 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.602728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.602796 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.603176 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.603180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.603444 (+ 264us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.604850 (+ 1406us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.604851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.604851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.605037 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.605197 (+ 160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.605198 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.605199 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.605253 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.605531 (+ 278us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.605532 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.605578 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.605637 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.605732 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":78,"threads_started":1}
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:04.627295 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35595
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:04.706277 18338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:04.706482 18338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:04.706504 18338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:04.707648 18338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:04.707684 18338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:04.707702 18338 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:04.707718 18338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:04.709071 18338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:35595
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18338
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:04.709255 18338 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:04.709430 18338 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:04.713891 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.713897 18346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:04.713897 18343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:04.714179 18338 server_base.cc:1034] running on GCE node
I20260923 23:10:04.714386 18338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:04.714612 18338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:04.715740 18338 hybrid_clock.cc:648] HybridClock initialized: now 1790205004715726 us; error 28 us; skew 500 ppm
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:04.718017 18338 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:04.718442 18338 webserver.cc:460] Webserver started at http://127.16.122.131:46637/ using document root <none> and password file <none>
I20260923 23:10:04.718641 18338 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:04.718686 18338 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:04.718803 18338 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:04.719825 18338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "c076a4eeca6b45cb91681e31382ab1cb"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "933826a946a97e2af6118e1a2ac435af"
server_key_iv: "ea892c4c70bbca57befd1d49a0e98fe5"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.720139 18338 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "c076a4eeca6b45cb91681e31382ab1cb"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "933826a946a97e2af6118e1a2ac435af"
server_key_iv: "ea892c4c70bbca57befd1d49a0e98fe5"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.721349 18338 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:10:04.721944 18353 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.722085 18338 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:04.722160 18338 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "c076a4eeca6b45cb91681e31382ab1cb"
format_stamp: "Formatted at 2026-09-23 23:10:04 on dist-test-slave-95hh"
server_key: "933826a946a97e2af6118e1a2ac435af"
server_key_iv: "ea892c4c70bbca57befd1d49a0e98fe5"
server_key_version: "encryptionkey@0"
I20260923 23:10:04.722224 18338 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:04.732865 18338 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:04.734490 18338 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:04.734612 18338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:04.735119 18338 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:04.735164 18338 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.735196 18338 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:04.735211 18338 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:04.741701 18338 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:34429
I20260923 23:10:04.741734 18468 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:34429 every 8 connection(s)
I20260923 23:10:04.742084 18338 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:04.744007 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.735422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:45233 (local address 127.16.122.190:35595)
0923 23:10:04.735468 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.735470 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.735727 (+ 257us) server_negotiation.cc:407] Connection header received
0923 23:10:04.736334 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.736335 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.736344 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.736380 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.737183 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.737683 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.738633 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.738823 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.738877 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.741612 (+ 2735us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.741615 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.741615 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.741622 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.742948 (+ 1326us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.743307 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.743308 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.743308 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.743373 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.743542 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.743542 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.743542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.743638 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.743680 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.743889 (+ 209us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.743945 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:10:04.744159 18362 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.735433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.131:45233)
0923 23:10:04.735591 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.735638 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.736240 (+ 602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.736470 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.736475 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.736823 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.737101 (+ 278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.737108 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.737762 (+ 654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.737762 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.738553 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.738555 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.738889 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.738890 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.739002 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.739468 (+ 466us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.739478 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.741503 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.743013 (+ 1510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.743016 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.743018 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.743249 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.743414 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.743415 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.743415 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.743501 (+ 86us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.743679 (+ 178us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.743682 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.743786 (+ 104us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.743869 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.743980 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
I20260923 23:10:04.744594 18469 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35595
I20260923 23:10:04.744668 18469 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:04.744822 18469 heartbeater.cc:511] Master 127.16.122.190:35595 requested a full tablet report, sending...
I20260923 23:10:04.745159 17985 ts_manager.cc:194] Registered new tserver with Master: c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:04.745620 17985 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:45233
I20260923 23:10:04.749961 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.745521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40075 (local address 127.16.122.190:35595)
0923 23:10:04.745569 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.745570 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.745618 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:10:04.745681 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.745682 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.745690 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.745731 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.746075 (+ 344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.746522 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.747218 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.747440 (+ 222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.747489 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.748238 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.748238 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.748239 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.748243 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.749193 (+ 950us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.749443 (+ 250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.749444 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.749444 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.749485 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.749608 (+ 123us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.749608 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.749609 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.749709 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.749759 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.749851 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.749902 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:04.749966 18475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.745455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.16.122.131:40075)
0923 23:10:04.745570 (+ 115us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.745579 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.745630 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.745767 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.745768 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.745945 (+ 177us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:04.746003 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.746005 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.746605 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.746606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.747157 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.747159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.747501 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.747502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.747556 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.747911 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:04.747915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:04.748162 (+ 247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:04.749248 (+ 1086us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.749249 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.749249 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.749393 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.749526 (+ 133us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:04.749527 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:04.749527 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:04.749569 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:04.749750 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:04.749751 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:04.749786 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.749834 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.749904 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":67,"threads_started":1}
I20260923 23:10:04.751740 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18338
I20260923 23:10:04.751822 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:04.751995 16874 external_mini_cluster.cc:1428] Setting key b9120c836c835400dc3ba43000ee1f85
I20260923 23:10:04.752707 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:10:04 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:04.773073 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.766681 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41564 (local address 127.16.122.190:35595)
0923 23:10:04.766731 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.766733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.766740 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:04.766778 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.766778 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.766791 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.766855 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:04.767389 (+ 534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.767871 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.768554 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.768764 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.768829 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.770911 (+ 2082us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:04.770913 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:04.770913 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:04.770921 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:04.772217 (+ 1296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.772545 (+ 328us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.772545 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.772546 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.772597 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:04.772711 (+ 114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:04.772711 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:04.772711 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:04.772807 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:04.772873 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.772949 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.773006 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:10:04.774499 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:04.776103 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:04.782132 18488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42329 (local address 127.0.0.1:34932)
0923 23:10:04.778253 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.778287 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.778340 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.778945 (+ 605us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.778946 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.778948 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:04.778985 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.778987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.780173 (+ 1186us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.780174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.781009 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.781011 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.781871 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781872 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.781926 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.781937 (+ 11us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.782005 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.782055 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":43,"threads_started":1}
I20260923 23:10:04.782164 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.16.122.129:42329)
0923 23:10:04.778635 (+ 138us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.778639 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.778648 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:04.778686 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.778688 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.778702 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.778775 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:04.779092 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.780109 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781073 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.781799 (+ 726us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781873 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.781932 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.782021 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.782062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":66,"threads_started":1}
I20260923 23:10:04.782267 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43046 (local address 127.16.122.130:42765)
0923 23:10:04.778496 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.778500 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.778721 (+ 221us) server_negotiation.cc:407] Connection header received
0923 23:10:04.778792 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.778795 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.778810 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.778881 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:04.779098 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.779791 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781235 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.781920 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781985 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.782031 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.782130 (+ 99us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.782171 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":70,"threads_started":1}
I20260923 23:10:04.782428 18487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778105 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42765 (local address 127.0.0.1:43046)
0923 23:10:04.778642 (+ 537us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.778652 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.778722 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.778931 (+ 209us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.778932 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.778934 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:04.778985 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.778987 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.779866 (+ 879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.779866 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.781155 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.781157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.781987 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.781988 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.782038 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.782041 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.782331 (+ 290us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.782375 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"mutex_wait_us":35,"thread_start_us":91,"threads_started":1}
I20260923 23:10:04.782858 18123 tablet_service.cc:1467] Processing CreateTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c (DEFAULT_TABLE table=test-table [id=d4b5da21f0164208a7bdc85463711bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.783201 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67084a0610e4a9ea3efe7e7dab2642c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.783524 18486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:34429 (local address 127.0.0.1:49690)
0923 23:10:04.778402 (+ 300us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.778410 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.778444 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.778627 (+ 183us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.778631 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.778633 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:04.778693 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.778695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.780012 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.780012 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.781160 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.781161 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.783228 (+ 2067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.783229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.783363 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.783367 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.783428 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.783464 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"mutex_wait_us":7,"thread_start_us":91,"threads_started":1}
I20260923 23:10:04.783537 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.778324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49690 (local address 127.16.122.131:34429)
0923 23:10:04.778488 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.778492 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.778501 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:04.778535 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.778537 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.778551 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.778613 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:04.778770 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.779918 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.782065 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.783140 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.783200 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.783252 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.783440 (+ 188us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.783470 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":76,"threads_started":1}
I20260923 23:10:04.784093 18403 tablet_service.cc:1467] Processing CreateTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c (DEFAULT_TABLE table=test-table [id=d4b5da21f0164208a7bdc85463711bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.784371 18403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67084a0610e4a9ea3efe7e7dab2642c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.786058 18492 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Bootstrap starting.
I20260923 23:10:04.786538 18263 tablet_service.cc:1467] Processing CreateTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c (DEFAULT_TABLE table=test-table [id=d4b5da21f0164208a7bdc85463711bfd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.786691 18492 tablet_bootstrap.cc:654] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.786652 18493 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Bootstrap starting.
I20260923 23:10:04.786877 18263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b67084a0610e4a9ea3efe7e7dab2642c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.787086 18492 log.cc:824] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:04.787572 18493 tablet_bootstrap.cc:654] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.787840 18493 log.cc:824] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:04.787847 18492 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: No bootstrap required, opened a new log
I20260923 23:10:04.787905 18492 ts_tablet_manager.cc:1397] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:10:04.788378 18493 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: No bootstrap required, opened a new log
I20260923 23:10:04.788431 18493 ts_tablet_manager.cc:1397] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:04.789829 18492 raft_consensus.cc:348] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.789873 18493 raft_consensus.cc:348] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.789989 18493 raft_consensus.cc:374] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.789991 18492 raft_consensus.cc:374] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.790017 18493 raft_consensus.cc:729] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c076a4eeca6b45cb91681e31382ab1cb, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.790026 18492 raft_consensus.cc:729] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b40d4071f61406187f038f775009ad1, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.790130 18493 consensus_queue.cc:260] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.790133 18492 consensus_queue.cc:260] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.790345 18493 ts_tablet_manager.cc:1428] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:04.790362 18492 ts_tablet_manager.cc:1428] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260923 23:10:04.790463 18469 heartbeater.cc:503] Master 127.16.122.190:35595 was elected leader, sending a full tablet report...
I20260923 23:10:04.790477 18189 heartbeater.cc:503] Master 127.16.122.190:35595 was elected leader, sending a full tablet report...
I20260923 23:10:04.790747 18496 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Bootstrap starting.
I20260923 23:10:04.791333 18496 tablet_bootstrap.cc:654] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.791636 18496 log.cc:824] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:04.792207 18496 tablet_bootstrap.cc:492] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: No bootstrap required, opened a new log
I20260923 23:10:04.792255 18496 ts_tablet_manager.cc:1397] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:04.793505 18496 raft_consensus.cc:348] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.793625 18496 raft_consensus.cc:374] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.793646 18496 raft_consensus.cc:729] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ea34b9211547ae8ebe7ba9e59894b5, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.793754 18496 consensus_queue.cc:260] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.793967 18496 ts_tablet_manager.cc:1428] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:04.794032 18329 heartbeater.cc:503] Master 127.16.122.190:35595 was elected leader, sending a full tablet report...
W20260923 23:10:04.847988 18330 tablet.cc:2369] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:04.862630 18498 raft_consensus.cc:482] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:04.862778 18498 raft_consensus.cc:504] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.863142 18498 leader_election.cc:288] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:04.865808 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.863389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50499 (local address 127.16.122.130:42765)
0923 23:10:04.863434 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.863436 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.863483 (+ 47us) server_negotiation.cc:407] Connection header received
0923 23:10:04.863509 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.863510 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.863517 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.863561 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:04.863716 (+ 155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.864174 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865079 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.865506 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865562 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.865571 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.865709 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.865744 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:10:04.865832 18082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.863334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42765 (local address 127.16.122.129:50499)
0923 23:10:04.863380 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.863391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.863433 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.863579 (+ 146us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.863580 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.863582 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:04.863654 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.863655 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.864252 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.864254 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.864995 (+ 741us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.864997 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.865579 (+ 582us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865580 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.865650 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.865653 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.865704 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.865739 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:10:04.865844 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.863392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:56471 (local address 127.16.122.131:34429)
0923 23:10:04.863455 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.863457 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.863618 (+ 161us) server_negotiation.cc:407] Connection header received
0923 23:10:04.863650 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.863651 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.863659 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.863705 (+ 46us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:04.863838 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.864271 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865140 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.865551 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865605 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.865614 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.865745 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.865779 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:10:04.866065 18501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.863334 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:34429 (local address 127.16.122.129:56471)
0923 23:10:04.863553 (+ 219us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:04.863561 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:04.863604 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:04.863714 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:04.863715 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:04.863716 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:04.863778 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.863780 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.864353 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.864354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.865060 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:04.865062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.865623 (+ 561us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.865624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:04.865687 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.865690 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:04.865733 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:04.865770 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":206,"mutex_wait_us":49,"thread_start_us":115,"threads_started":1}
I20260923 23:10:04.866184 18283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67084a0610e4a9ea3efe7e7dab2642c" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" is_pre_election: true
I20260923 23:10:04.866341 18283 raft_consensus.cc:2454] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 0.
I20260923 23:10:04.866413 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67084a0610e4a9ea3efe7e7dab2642c" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb" is_pre_election: true
I20260923 23:10:04.866542 18423 raft_consensus.cc:2454] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 0.
I20260923 23:10:04.866585 18075 leader_election.cc:302] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b40d4071f61406187f038f775009ad1, 72ea34b9211547ae8ebe7ba9e59894b5; no voters:
I20260923 23:10:04.866693 18498 raft_consensus.cc:2781] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:04.866750 18498 raft_consensus.cc:482] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:04.866775 18498 raft_consensus.cc:3037] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:04.867691 18498 raft_consensus.cc:504] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.867807 18498 leader_election.cc:288] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:04.867945 18283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67084a0610e4a9ea3efe7e7dab2642c" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ea34b9211547ae8ebe7ba9e59894b5"
I20260923 23:10:04.867938 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b67084a0610e4a9ea3efe7e7dab2642c" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb"
I20260923 23:10:04.868016 18283 raft_consensus.cc:3037] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:04.868016 18423 raft_consensus.cc:3037] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:04.868659 18283 raft_consensus.cc:2454] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 1.
I20260923 23:10:04.868656 18423 raft_consensus.cc:2454] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 1.
I20260923 23:10:04.868803 18076 leader_election.cc:302] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b40d4071f61406187f038f775009ad1, c076a4eeca6b45cb91681e31382ab1cb; no voters:
I20260923 23:10:04.868881 18498 raft_consensus.cc:2781] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:04.869040 18498 raft_consensus.cc:686] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 2b40d4071f61406187f038f775009ad1, State: Running, Role: LEADER
I20260923 23:10:04.869161 18498 consensus_queue.cc:237] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.869946 17985 catalog_manager.cc:5495] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b40d4071f61406187f038f775009ad1 (127.16.122.129). New cstate: current_term: 1 leader_uuid: "2b40d4071f61406187f038f775009ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:04.897047 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:04.894667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34946 (local address 127.16.122.129:42329)
0923 23:10:04.894718 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:04.894720 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:04.894732 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:04.894838 (+ 106us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:04.894839 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:04.894856 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:04.894906 (+ 50us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:04.895057 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.895514 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.896260 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:04.896447 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:04.896502 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:04.896703 (+ 201us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:04.896798 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:04.896873 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:04.896921 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 23:10:04.896946 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:10:04.898897 18283 raft_consensus.cc:1261] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Refusing update from remote peer 2b40d4071f61406187f038f775009ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:04.898897 18423 raft_consensus.cc:1261] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Refusing update from remote peer 2b40d4071f61406187f038f775009ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:04.899258 18498 consensus_queue.cc:1035] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:04.899390 18502 consensus_queue.cc:1035] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:04.901878 18506 mvcc.cc:205] Tried to move back new op lower bound from 7332679700062973952 to 7332679699944615936. Current Snapshot: MvccSnapshot[applied={T|T < 7332679700062973952}]
I20260923 23:10:04.905354 17985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:04.905436 17985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:04.906113 17985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Sending DeleteTablet for 3 replicas of tablet b67084a0610e4a9ea3efe7e7dab2642c
I20260923 23:10:04.906389 18263 tablet_service.cc:1514] Processing DeleteTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:04 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:43046
I20260923 23:10:04.906412 18123 tablet_service.cc:1514] Processing DeleteTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:04 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:34932
I20260923 23:10:04.906662 18516 tablet_replica.cc:330] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: stopping tablet replica
I20260923 23:10:04.906795 18516 raft_consensus.cc:2229] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:04.906812 18403 tablet_service.cc:1514] Processing DeleteTablet for tablet b67084a0610e4a9ea3efe7e7dab2642c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:04 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49690
I20260923 23:10:04.907023 18516 raft_consensus.cc:2258] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:04.907050 18517 tablet_replica.cc:330] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: stopping tablet replica
I20260923 23:10:04.907006 17985 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 23:10:04.907145 18517 raft_consensus.cc:2229] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Raft consensus shutting down.
W20260923 23:10:04.907167 17985 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:04.907243 18517 raft_consensus.cc:2258] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:04.907721 18517 ts_tablet_manager.cc:1905] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:04.907721 18516 ts_tablet_manager.cc:1905] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:04.908268 18123 tablet_service.cc:1467] Processing CreateTablet for tablet 190d8fb70f454c4f8893d3f81577b969 (DEFAULT_TABLE table=test-table [id=a754167d66ca4897bbb1146b0829e225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.908332 18263 tablet_service.cc:1467] Processing CreateTablet for tablet 190d8fb70f454c4f8893d3f81577b969 (DEFAULT_TABLE table=test-table [id=a754167d66ca4897bbb1146b0829e225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.908416 18263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190d8fb70f454c4f8893d3f81577b969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.908391 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190d8fb70f454c4f8893d3f81577b969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.909668 18492 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Bootstrap starting.
I20260923 23:10:04.909698 18515 tablet_replica.cc:330] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: stopping tablet replica
I20260923 23:10:04.909788 18515 raft_consensus.cc:2229] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:04.909889 18515 raft_consensus.cc:2258] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:04.909907 18516 ts_tablet_manager.cc:1918] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:04.910028 18517 ts_tablet_manager.cc:1918] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:04.910068 18492 tablet_bootstrap.cc:654] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.910091 18517 log.cc:1196] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/b67084a0610e4a9ea3efe7e7dab2642c
I20260923 23:10:04.910058 18516 log.cc:1196] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/b67084a0610e4a9ea3efe7e7dab2642c
I20260923 23:10:04.910326 18516 ts_tablet_manager.cc:1939] T b67084a0610e4a9ea3efe7e7dab2642c P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting consensus metadata
I20260923 23:10:04.910336 18517 ts_tablet_manager.cc:1939] T b67084a0610e4a9ea3efe7e7dab2642c P c076a4eeca6b45cb91681e31382ab1cb: Deleting consensus metadata
I20260923 23:10:04.910408 18496 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Bootstrap starting.
I20260923 23:10:04.910771 18496 tablet_bootstrap.cc:654] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.910918 18492 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: No bootstrap required, opened a new log
I20260923 23:10:04.910959 18492 ts_tablet_manager.cc:1397] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:04.911113 18492 raft_consensus.cc:348] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.911140 17971 catalog_manager.cc:4841] TS 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765): tablet b67084a0610e4a9ea3efe7e7dab2642c (table test-table [id=d4b5da21f0164208a7bdc85463711bfd]) successfully deleted
I20260923 23:10:04.911183 18492 raft_consensus.cc:374] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.911201 18492 raft_consensus.cc:729] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b40d4071f61406187f038f775009ad1, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.911183 17972 catalog_manager.cc:4841] TS c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429): tablet b67084a0610e4a9ea3efe7e7dab2642c (table test-table [id=d4b5da21f0164208a7bdc85463711bfd]) successfully deleted
I20260923 23:10:04.911243 18492 consensus_queue.cc:260] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.911332 18492 ts_tablet_manager.cc:1428] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.911376 18403 tablet_service.cc:1467] Processing CreateTablet for tablet 190d8fb70f454c4f8893d3f81577b969 (DEFAULT_TABLE table=test-table [id=a754167d66ca4897bbb1146b0829e225]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:04.911451 18403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 190d8fb70f454c4f8893d3f81577b969. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:04.911568 18515 ts_tablet_manager.cc:1905] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:04.912391 18493 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Bootstrap starting.
I20260923 23:10:04.912498 18496 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: No bootstrap required, opened a new log
I20260923 23:10:04.912534 18496 ts_tablet_manager.cc:1397] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:04.912640 18496 raft_consensus.cc:348] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.912681 18496 raft_consensus.cc:374] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.912695 18496 raft_consensus.cc:729] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ea34b9211547ae8ebe7ba9e59894b5, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.912730 18496 consensus_queue.cc:260] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.912765 18493 tablet_bootstrap.cc:654] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:04.912802 18496 ts_tablet_manager.cc:1428] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:04.912894 18515 ts_tablet_manager.cc:1918] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:04.912952 18515 log.cc:1196] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/b67084a0610e4a9ea3efe7e7dab2642c
I20260923 23:10:04.913172 18515 ts_tablet_manager.cc:1939] T b67084a0610e4a9ea3efe7e7dab2642c P 2b40d4071f61406187f038f775009ad1: Deleting consensus metadata
I20260923 23:10:04.913583 17970 catalog_manager.cc:4841] TS 2b40d4071f61406187f038f775009ad1 (127.16.122.129:42329): tablet b67084a0610e4a9ea3efe7e7dab2642c (table test-table [id=d4b5da21f0164208a7bdc85463711bfd]) successfully deleted
I20260923 23:10:04.913702 18493 tablet_bootstrap.cc:492] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: No bootstrap required, opened a new log
I20260923 23:10:04.913739 18493 ts_tablet_manager.cc:1397] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:04.913856 18493 raft_consensus.cc:348] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.913897 18493 raft_consensus.cc:374] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:04.913908 18493 raft_consensus.cc:729] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c076a4eeca6b45cb91681e31382ab1cb, State: Initialized, Role: FOLLOWER
I20260923 23:10:04.913937 18493 consensus_queue.cc:260] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:04.913991 18493 ts_tablet_manager.cc:1428] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
W20260923 23:10:04.941708 18190 tablet.cc:2369] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:04.992868 18470 tablet.cc:2369] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:05.074102 18512 raft_consensus.cc:482] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:05.074281 18512 raft_consensus.cc:504] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.074486 18512 leader_election.cc:288] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:05.074669 18283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190d8fb70f454c4f8893d3f81577b969" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" is_pre_election: true
I20260923 23:10:05.074772 18283 raft_consensus.cc:2454] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 0.
I20260923 23:10:05.074741 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190d8fb70f454c4f8893d3f81577b969" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb" is_pre_election: true
I20260923 23:10:05.074822 18423 raft_consensus.cc:2454] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 0.
I20260923 23:10:05.074956 18076 leader_election.cc:302] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b40d4071f61406187f038f775009ad1, c076a4eeca6b45cb91681e31382ab1cb; no voters:
I20260923 23:10:05.075064 18512 raft_consensus.cc:2781] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:05.075121 18512 raft_consensus.cc:482] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:05.075134 18512 raft_consensus.cc:3037] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.075824 18512 raft_consensus.cc:504] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.075941 18512 leader_election.cc:288] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 election: Requested vote from peers 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:05.076107 18283 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190d8fb70f454c4f8893d3f81577b969" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "72ea34b9211547ae8ebe7ba9e59894b5"
I20260923 23:10:05.076141 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "190d8fb70f454c4f8893d3f81577b969" candidate_uuid: "2b40d4071f61406187f038f775009ad1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb"
I20260923 23:10:05.076191 18283 raft_consensus.cc:3037] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.076200 18423 raft_consensus.cc:3037] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.076887 18283 raft_consensus.cc:2454] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 1.
I20260923 23:10:05.076889 18423 raft_consensus.cc:2454] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2b40d4071f61406187f038f775009ad1 in term 1.
I20260923 23:10:05.077052 18075 leader_election.cc:302] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2b40d4071f61406187f038f775009ad1, 72ea34b9211547ae8ebe7ba9e59894b5; no voters:
I20260923 23:10:05.077150 18512 raft_consensus.cc:2781] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:05.077226 18512 raft_consensus.cc:686] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 1 LEADER]: Becoming Leader. State: Replica: 2b40d4071f61406187f038f775009ad1, State: Running, Role: LEADER
I20260923 23:10:05.077283 18512 consensus_queue.cc:237] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.077814 17986 catalog_manager.cc:5495] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2b40d4071f61406187f038f775009ad1 (127.16.122.129). New cstate: current_term: 1 leader_uuid: "2b40d4071f61406187f038f775009ad1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:10:05 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:05.113518 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.107736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41576 (local address 127.16.122.190:35595)
0923 23:10:05.107820 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.107822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.107870 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:10:05.107919 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.107920 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.107931 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.107980 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:05.108360 (+ 380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.108831 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.109447 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.109611 (+ 164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.109651 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.111539 (+ 1888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:05.111540 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:05.111540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:05.111546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:05.112738 (+ 1192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.113009 (+ 271us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.113010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.113010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.113053 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.113163 (+ 110us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.113163 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.113164 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.113252 (+ 88us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:05.113293 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.113393 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.113454 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:10:05.113593 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.107672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35595 (local address 127.0.0.1:41576)
0923 23:10:05.107800 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.107810 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.107876 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.107985 (+ 109us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.107985 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.108265 (+ 280us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:05.108311 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.108313 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.108881 (+ 568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.108882 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.109397 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.109398 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.109655 (+ 257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.109655 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.109689 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.110111 (+ 422us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:05.110115 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:05.111459 (+ 1344us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:05.112794 (+ 1335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:05.112798 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:05.112800 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:05.112961 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:05.113087 (+ 126us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:05.113087 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:05.113088 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:05.113132 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:05.113292 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:05.113296 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:05.113333 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.113378 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.113499 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"mutex_wait_us":44,"thread_start_us":61,"threads_started":1}
I20260923 23:10:05.115294 17986 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:41576:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:10:05.115485 17986 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:05.116778 18263 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f1276453944f99b1f43a1492fb8a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90137fbcb755437ca060e301c56788fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:05.116787 18123 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f1276453944f99b1f43a1492fb8a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90137fbcb755437ca060e301c56788fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:05.116931 18263 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f1276453944f99b1f43a1492fb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:05.116931 18123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f1276453944f99b1f43a1492fb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:05.116954 18403 tablet_service.cc:1467] Processing CreateTablet for tablet fb8f1276453944f99b1f43a1492fb8a5 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=90137fbcb755437ca060e301c56788fe]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:05.117098 18403 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fb8f1276453944f99b1f43a1492fb8a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:05.118317 18496 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: Bootstrap starting.
I20260923 23:10:05.118346 18493 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: Bootstrap starting.
I20260923 23:10:05.118677 18496 tablet_bootstrap.cc:654] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:05.118706 18493 tablet_bootstrap.cc:654] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:05.119005 18492 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: Bootstrap starting.
I20260923 23:10:05.119309 18492 tablet_bootstrap.cc:654] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:05.119402 18493 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: No bootstrap required, opened a new log
I20260923 23:10:05.119450 18493 ts_tablet_manager.cc:1397] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:05.119597 18493 raft_consensus.cc:348] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.119675 18493 raft_consensus.cc:374] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:05.119695 18493 raft_consensus.cc:729] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c076a4eeca6b45cb91681e31382ab1cb, State: Initialized, Role: FOLLOWER
I20260923 23:10:05.119737 18493 consensus_queue.cc:260] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.119800 18496 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: No bootstrap required, opened a new log
I20260923 23:10:05.119827 18493 ts_tablet_manager.cc:1428] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:05.119848 18496 ts_tablet_manager.cc:1397] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:05.119906 18497 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.119982 18496 raft_consensus.cc:348] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.120007 18497 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.120046 18496 raft_consensus.cc:374] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:05.120072 18496 raft_consensus.cc:729] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72ea34b9211547ae8ebe7ba9e59894b5, State: Initialized, Role: FOLLOWER
I20260923 23:10:05.120116 18496 consensus_queue.cc:260] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.120209 18496 ts_tablet_manager.cc:1428] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:05.120251 18500 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.120342 18500 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.120395 18492 tablet_bootstrap.cc:492] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: No bootstrap required, opened a new log
I20260923 23:10:05.120445 18492 ts_tablet_manager.cc:1397] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:05.120554 18492 raft_consensus.cc:348] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.120605 18492 raft_consensus.cc:374] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:05.120625 18492 raft_consensus.cc:729] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2b40d4071f61406187f038f775009ad1, State: Initialized, Role: FOLLOWER
I20260923 23:10:05.120669 18492 consensus_queue.cc:260] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.120756 18492 ts_tablet_manager.cc:1428] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:05.120800 18512 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.120877 18512 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.134648 18500 raft_consensus.cc:482] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:05.134764 18500 raft_consensus.cc:504] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.135121 18500 leader_election.cc:288] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2b40d4071f61406187f038f775009ad1 (127.16.122.129:42329), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:05.137945 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.135285 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:49353 (local address 127.16.122.129:42329)
0923 23:10:05.135354 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.135356 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.135518 (+ 162us) server_negotiation.cc:407] Connection header received
0923 23:10:05.135545 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.135546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.135558 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.135598 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:05.135771 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.136249 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137260 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.137619 (+ 359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137681 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.137689 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.137846 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.137880 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:10:05.137940 18489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.135485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:46653 (local address 127.16.122.131:34429)
0923 23:10:05.135541 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.135543 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.135549 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:10:05.135570 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.135571 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.135578 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.135629 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:05.135773 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.136246 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137265 (+ 1019us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.137614 (+ 349us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137672 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.137682 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.137829 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.137869 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:10:05.137985 18531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.135228 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42329 (local address 127.16.122.130:49353)
0923 23:10:05.135359 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.135370 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.135434 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.135619 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.135620 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.135623 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:05.135712 (+ 89us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.135714 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.136324 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.136325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.137177 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.137179 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.137699 (+ 520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.137782 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.137785 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.137836 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.137873 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"mutex_wait_us":44,"thread_start_us":72,"threads_started":1}
I20260923 23:10:05.137992 18532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.135289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:34429 (local address 127.16.122.130:46653)
0923 23:10:05.135389 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.135398 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.135451 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.135639 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.135640 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.135642 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:05.135712 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.135713 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.136326 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.136327 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.137178 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.137180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.137698 (+ 518us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.137699 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.137766 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.137769 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.137811 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.137853 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"mutex_wait_us":34,"thread_start_us":64,"threads_started":1}
I20260923 23:10:05.138242 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f1276453944f99b1f43a1492fb8a5" candidate_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb" is_pre_election: true
I20260923 23:10:05.138267 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f1276453944f99b1f43a1492fb8a5" candidate_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b40d4071f61406187f038f775009ad1" is_pre_election: true
I20260923 23:10:05.138332 18423 raft_consensus.cc:2454] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ea34b9211547ae8ebe7ba9e59894b5 in term 0.
I20260923 23:10:05.138383 18143 raft_consensus.cc:2454] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 72ea34b9211547ae8ebe7ba9e59894b5 in term 0.
I20260923 23:10:05.138496 18216 leader_election.cc:302] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ea34b9211547ae8ebe7ba9e59894b5, c076a4eeca6b45cb91681e31382ab1cb; no voters:
I20260923 23:10:05.138639 18500 raft_consensus.cc:2781] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:05.138671 18500 raft_consensus.cc:482] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:05.138690 18500 raft_consensus.cc:3037] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.139406 18500 raft_consensus.cc:504] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.139541 18500 leader_election.cc:288] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [CANDIDATE]: Term 1 election: Requested vote from peers 2b40d4071f61406187f038f775009ad1 (127.16.122.129:42329), c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429)
I20260923 23:10:05.139655 18143 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f1276453944f99b1f43a1492fb8a5" candidate_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2b40d4071f61406187f038f775009ad1"
I20260923 23:10:05.139693 18423 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fb8f1276453944f99b1f43a1492fb8a5" candidate_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c076a4eeca6b45cb91681e31382ab1cb"
I20260923 23:10:05.139739 18143 raft_consensus.cc:3037] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.139765 18423 raft_consensus.cc:3037] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:05.140446 18423 raft_consensus.cc:2454] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ea34b9211547ae8ebe7ba9e59894b5 in term 1.
I20260923 23:10:05.140527 18143 raft_consensus.cc:2454] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 72ea34b9211547ae8ebe7ba9e59894b5 in term 1.
I20260923 23:10:05.140607 18216 leader_election.cc:302] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 72ea34b9211547ae8ebe7ba9e59894b5, c076a4eeca6b45cb91681e31382ab1cb; no voters:
I20260923 23:10:05.140709 18500 raft_consensus.cc:2781] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:05.140829 18500 raft_consensus.cc:686] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 LEADER]: Becoming Leader. State: Replica: 72ea34b9211547ae8ebe7ba9e59894b5, State: Running, Role: LEADER
I20260923 23:10:05.140897 18500 consensus_queue.cc:237] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } }
I20260923 23:10:05.141215 18533 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: TxnStatusTablet state changed. Reason: New leader 72ea34b9211547ae8ebe7ba9e59894b5. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.141319 18533 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:05.141489 18535 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:10:05.141582 18535 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:10:05.141546 17985 catalog_manager.cc:5495] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 reported cstate change: term changed from 0 to 1, leader changed from <none> to 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:05.150158 18527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.147883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42765 (local address 127.0.0.1:43048)
0923 23:10:05.147944 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.147956 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.148000 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.148153 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.148154 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.148155 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:05.148201 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.148203 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.148784 (+ 581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.148785 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.149354 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.149356 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.149683 (+ 327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.149684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.149751 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.149765 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.149997 (+ 232us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.150004 (+ 7us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.150046 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.150094 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:10:05.150141 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.147962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43048 (local address 127.16.122.130:42765)
0923 23:10:05.148006 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.148008 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.148016 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:05.148075 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.148076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.148093 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.148141 (+ 48us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:05.148244 (+ 103us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.148715 (+ 471us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.149428 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.149627 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.149680 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.149843 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.149945 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.150004 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.150048 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.150069 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:10:05.151247 18143 raft_consensus.cc:1261] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Refusing update from remote peer 72ea34b9211547ae8ebe7ba9e59894b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:05.151424 18423 raft_consensus.cc:1261] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Refusing update from remote peer 72ea34b9211547ae8ebe7ba9e59894b5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:05.151650 18533 consensus_queue.cc:1035] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:05.151777 18500 consensus_queue.cc:1035] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:05.152793 18512 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: TxnStatusTablet state changed. Reason: New leader 72ea34b9211547ae8ebe7ba9e59894b5. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.152894 18512 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.153290 18497 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: TxnStatusTablet state changed. Reason: New leader 72ea34b9211547ae8ebe7ba9e59894b5. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.153374 18497 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.153505 18500 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.153578 18500 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:05.153767 18508 mvcc.cc:205] Tried to move back new op lower bound from 7332679701097340928 to 7332679701057728512. Current Snapshot: MvccSnapshot[applied={T|T < 7332679701097340928 or (T in {7332679701097340928})}]
I20260923 23:10:05.154163 18497 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.154244 18497 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P c076a4eeca6b45cb91681e31382ab1cb: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.154232 18505 mvcc.cc:205] Tried to move back new op lower bound from 7332679701097340928 to 7332679701057728512. Current Snapshot: MvccSnapshot[applied={T|T < 7332679701097340928}]
I20260923 23:10:05.154253 18500 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.154325 18500 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 72ea34b9211547ae8ebe7ba9e59894b5: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:05.154309 18512 tablet_replica.cc:439] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2b40d4071f61406187f038f775009ad1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42329 } } peers { permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 } } peers { permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 } } }
I20260923 23:10:05.154369 18512 tablet_replica.cc:442] T fb8f1276453944f99b1f43a1492fb8a5 P 2b40d4071f61406187f038f775009ad1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:05.159925 18490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.157774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44621 (local address 127.16.122.130:42765)
0923 23:10:05.157846 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.157848 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.157900 (+ 52us) server_negotiation.cc:407] Connection header received
0923 23:10:05.157958 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.157959 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.157968 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.158010 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:05.158103 (+ 93us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.158555 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.159332 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.159522 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.159573 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.159714 (+ 141us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.159744 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.159783 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.159841 (+ 58us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.159858 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:10:05.159952 18543 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.157713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42765 (local address 127.16.122.129:44621)
0923 23:10:05.157851 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.157861 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.157914 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.158011 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.158012 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.158013 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:05.158060 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.158062 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.158618 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.158619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.159258 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.159260 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.159576 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.159578 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.159635 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.159649 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.159789 (+ 140us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.159791 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.159836 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.159886 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":70,"threads_started":1}
I20260923 23:10:05.161877 18423 raft_consensus.cc:1261] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Refusing update from remote peer 2b40d4071f61406187f038f775009ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:05.161923 18283 raft_consensus.cc:1261] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Refusing update from remote peer 2b40d4071f61406187f038f775009ad1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:05.162091 18512 consensus_queue.cc:1035] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "72ea34b9211547ae8ebe7ba9e59894b5" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42765 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:05.162269 18512 consensus_queue.cc:1035] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c076a4eeca6b45cb91681e31382ab1cb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:05.169955 18548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.167149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42329 (local address 127.16.122.130:54993)
0923 23:10:05.167400 (+ 251us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:05.167410 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:05.167476 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:05.167677 (+ 201us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:05.167678 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:05.167680 (+ 2us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:05.167746 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.167748 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.168618 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.168619 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.169194 (+ 575us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:05.169196 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.169624 (+ 428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.169625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:05.169668 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.169681 (+ 13us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.169822 (+ 141us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.169823 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:05.169852 (+ 29us) client_negotiation.cc:240] Negotiation successful
0923 23:10:05.169883 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":73,"threads_started":1}
I20260923 23:10:05.169956 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.167207 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:54993 (local address 127.16.122.129:42329)
0923 23:10:05.167250 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.167252 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.167458 (+ 206us) server_negotiation.cc:407] Connection header received
0923 23:10:05.167523 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.167524 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.167533 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.167582 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:05.167816 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.168551 (+ 735us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.169261 (+ 710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.169564 (+ 303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.169628 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.169742 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:05.169776 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:05.169820 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.169867 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.169886 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:05.176589 17985 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:05.176709 17985 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41564:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:05.177220 17985 catalog_manager.cc:5782] T 00000000000000000000000000000000 P fd2e09af0b1d447ba23bdc8020e3b727: Sending DeleteTablet for 3 replicas of tablet 190d8fb70f454c4f8893d3f81577b969
I20260923 23:10:05.177429 18263 tablet_service.cc:1514] Processing DeleteTablet for tablet 190d8fb70f454c4f8893d3f81577b969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:05 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:43046
I20260923 23:10:05.177492 18403 tablet_service.cc:1514] Processing DeleteTablet for tablet 190d8fb70f454c4f8893d3f81577b969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:05 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49690
I20260923 23:10:05.177541 18516 tablet_replica.cc:330] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: stopping tablet replica
I20260923 23:10:05.177583 18517 tablet_replica.cc:330] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: stopping tablet replica
I20260923 23:10:05.177584 18516 raft_consensus.cc:2229] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:05.177625 18517 raft_consensus.cc:2229] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:05.177634 18516 raft_consensus.cc:2258] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:05.177661 18517 raft_consensus.cc:2258] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:05.177856 18123 tablet_service.cc:1514] Processing DeleteTablet for tablet 190d8fb70f454c4f8893d3f81577b969 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:05 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:34932
I20260923 23:10:05.177935 18515 tablet_replica.cc:330] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: stopping tablet replica
I20260923 23:10:05.177979 18515 raft_consensus.cc:2229] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:05.178032 18515 raft_consensus.cc:2258] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:05.178305 18517 ts_tablet_manager.cc:1905] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:05.178325 18516 ts_tablet_manager.cc:1905] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:05.178346 18515 ts_tablet_manager.cc:1905] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:05.180055 18516 ts_tablet_manager.cc:1918] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:05.180055 18517 ts_tablet_manager.cc:1918] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:05.180107 18516 log.cc:1196] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/190d8fb70f454c4f8893d3f81577b969
I20260923 23:10:05.180107 18517 log.cc:1196] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/190d8fb70f454c4f8893d3f81577b969
I20260923 23:10:05.180298 18517 ts_tablet_manager.cc:1939] T 190d8fb70f454c4f8893d3f81577b969 P c076a4eeca6b45cb91681e31382ab1cb: Deleting consensus metadata
I20260923 23:10:05.180298 18516 ts_tablet_manager.cc:1939] T 190d8fb70f454c4f8893d3f81577b969 P 72ea34b9211547ae8ebe7ba9e59894b5: Deleting consensus metadata
I20260923 23:10:05.180476 18515 ts_tablet_manager.cc:1918] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:05.180532 18515 log.cc:1196] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/190d8fb70f454c4f8893d3f81577b969
I20260923 23:10:05.180703 18515 ts_tablet_manager.cc:1939] T 190d8fb70f454c4f8893d3f81577b969 P 2b40d4071f61406187f038f775009ad1: Deleting consensus metadata
I20260923 23:10:05.180730 17971 catalog_manager.cc:4841] TS 72ea34b9211547ae8ebe7ba9e59894b5 (127.16.122.130:42765): tablet 190d8fb70f454c4f8893d3f81577b969 (table test-table [id=a754167d66ca4897bbb1146b0829e225]) successfully deleted
I20260923 23:10:05.180974 17970 catalog_manager.cc:4841] TS 2b40d4071f61406187f038f775009ad1 (127.16.122.129:42329): tablet 190d8fb70f454c4f8893d3f81577b969 (table test-table [id=a754167d66ca4897bbb1146b0829e225]) successfully deleted
I20260923 23:10:05.181243 17972 catalog_manager.cc:4841] TS c076a4eeca6b45cb91681e31382ab1cb (127.16.122.131:34429): tablet 190d8fb70f454c4f8893d3f81577b969 (table test-table [id=a754167d66ca4897bbb1146b0829e225]) successfully deleted
Sep 23 23:10:05 dist-test-slave-95hh krb5kdc[17941](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205004, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:05.188828 18491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.180057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34962 (local address 127.16.122.129:42329)
0923 23:10:05.180096 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.180098 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.180572 (+ 474us) server_negotiation.cc:407] Connection header received
0923 23:10:05.180730 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.180731 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.180744 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.180773 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:05.181482 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.182242 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.182895 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.183220 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.183286 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.186013 (+ 2727us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:05.186023 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:05.186023 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:05.186030 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:05.187782 (+ 1752us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.188074 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.188078 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.188079 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.188140 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.188284 (+ 144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.188285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.188285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.188457 (+ 172us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:05.188595 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.188630 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.188715 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
W20260923 23:10:05.189077 18183 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34962
I20260923 23:10:05.194983 18042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:05.189917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41592 (local address 127.16.122.190:35595)
0923 23:10:05.189964 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:05.189966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:05.190011 (+ 45us) server_negotiation.cc:407] Connection header received
0923 23:10:05.190071 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:05.190072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:05.190082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:05.190133 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:05.190559 (+ 426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.191063 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.191697 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:05.191887 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:05.191939 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:05.192928 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:05.192929 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:05.192930 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:05.192936 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:05.194059 (+ 1123us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.194334 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.194335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.194336 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.194386 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:05.194546 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:05.194547 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:05.194548 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:05.194650 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:05.194731 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:05.194822 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 23:10:05.194893 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:10:05.195222 17985 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:41592
I20260923 23:10:05.195749 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18058
I20260923 23:10:05.203065 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18198
I20260923 23:10:05.207401 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18338
I20260923 23:10:05.211822 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 17954
2026-09-23T23:10:05Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (2143 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:05 dist-test-slave-95hh krb5kdc[18556](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:05 dist-test-slave-95hh krb5kdc[18556](info): set up 2 sockets
Sep 23 23:10:05 dist-test-slave-95hh krb5kdc[18556](info): commencing operation
krb5kdc: starting...
W20260923 23:10:06.161952 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:06.271679 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36229
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:36229
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:06.351217 18569 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:06.351557 18569 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:06.351594 18569 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:06.353451 18569 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:06.353518 18569 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:10:06.353538 18569 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:06.353566 18569 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:06.353590 18569 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:06.353612 18569 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:06.355829 18569 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:36229
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36229
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18569
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:06.356115 18569 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:06.356385 18569 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:06.361069 18574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:06.361085 18575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.361107 18569 server_base.cc:1034] running on GCE node
W20260923 23:10:06.361073 18577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.361487 18569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:06.361753 18569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:06.362885 18569 hybrid_clock.cc:648] HybridClock initialized: now 1790205006362878 us; error 33 us; skew 500 ppm
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:06.365252 18569 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:06.365679 18569 webserver.cc:460] Webserver started at http://127.16.122.190:37859/ using document root <none> and password file <none>
I20260923 23:10:06.365875 18569 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:06.365917 18569 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:06.366034 18569 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:06.367103 18569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "5f0539a70f3c47e09a93157b7999bc8b"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "80db92841cc56cb831e61e90dd966984"
server_key_iv: "0d896338cc34dad440917159563c25f8"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.367455 18569 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "5f0539a70f3c47e09a93157b7999bc8b"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "80db92841cc56cb831e61e90dd966984"
server_key_iv: "0d896338cc34dad440917159563c25f8"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.368633 18569 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:06.369230 18584 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.369371 18569 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:06.369451 18569 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "5f0539a70f3c47e09a93157b7999bc8b"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "80db92841cc56cb831e61e90dd966984"
server_key_iv: "0d896338cc34dad440917159563c25f8"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.369508 18569 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:06.377144 18569 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:06.378719 18569 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:06.378839 18569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:06.382999 18569 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:36229
I20260923 23:10:06.383082 18646 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:36229 every 8 connection(s)
I20260923 23:10:06.383360 18569 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:06.383903 18647 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:06.385915 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18569
I20260923 23:10:06.385999 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:06.386171 16874 external_mini_cluster.cc:1428] Setting key aaf1b8ae36ef46921bcc34baf7bc43ae
I20260923 23:10:06.386852 18647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Bootstrap starting.
I20260923 23:10:06.387555 18647 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:06.387858 18647 log.cc:824] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:06.388562 18647 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: No bootstrap required, opened a new log
I20260923 23:10:06.390638 18647 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } }
I20260923 23:10:06.390796 18647 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:06.390847 18647 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5f0539a70f3c47e09a93157b7999bc8b, State: Initialized, Role: FOLLOWER
I20260923 23:10:06.390973 18647 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } }
I20260923 23:10:06.391063 18647 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:06.391103 18647 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:06.391149 18647 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:06.391762 18647 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } }
I20260923 23:10:06.391880 18647 leader_election.cc:302] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5f0539a70f3c47e09a93157b7999bc8b; no voters:
I20260923 23:10:06.392040 18647 leader_election.cc:288] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:06.392077 18652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:06.392215 18652 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [term 1 LEADER]: Becoming Leader. State: Replica: 5f0539a70f3c47e09a93157b7999bc8b, State: Running, Role: LEADER
I20260923 23:10:06.392295 18647 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:06.392302 18652 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } }
I20260923 23:10:06.392616 18652 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5f0539a70f3c47e09a93157b7999bc8b. Latest consensus state: current_term: 1 leader_uuid: "5f0539a70f3c47e09a93157b7999bc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } } }
I20260923 23:10:06.392675 18652 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:06.392841 18654 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5f0539a70f3c47e09a93157b7999bc8b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5f0539a70f3c47e09a93157b7999bc8b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36229 } } }
I20260923 23:10:06.392973 18654 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:06.393110 18656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:06.393676 18656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:06.395640 18656 catalog_manager.cc:1347] Generated new cluster ID: 7d584518271749bc89fa241c7351250d
I20260923 23:10:06.395689 18656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:06.397126 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.386675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43792 (local address 127.16.122.190:36229)
0923 23:10:06.386855 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.386874 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.387222 (+ 348us) server_negotiation.cc:407] Connection header received
0923 23:10:06.387861 (+ 639us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.387870 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.387932 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.388054 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.388838 (+ 784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.389685 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.390698 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.391230 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.391317 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.393866 (+ 2549us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.393870 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.393870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.393882 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.395765 (+ 1883us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.396069 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.396072 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.396073 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.396160 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.396334 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.396335 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.396335 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.396561 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.396692 (+ 131us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.396799 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.396925 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260923 23:10:06.414062 18656 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:06.414563 18656 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:06.417248 18656 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Generated new TSK 0
I20260923 23:10:06.417418 18656 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:06.441668 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:36229
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:06.532236 18673 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:06.532454 18673 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:06.532488 18673 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:06.533715 18673 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:06.533769 18673 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:06.533798 18673 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:10:06.533813 18673 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:06.533838 18673 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:06.535496 18673 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18673
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:06.535739 18673 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:06.535989 18673 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:06.541111 18681 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:06.541155 18679 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.541177 18673 server_base.cc:1034] running on GCE node
W20260923 23:10:06.541121 18678 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.541539 18673 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:06.541811 18673 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:06.542976 18673 hybrid_clock.cc:648] HybridClock initialized: now 1790205006542954 us; error 30 us; skew 500 ppm
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:06.545497 18673 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:06.545920 18673 webserver.cc:460] Webserver started at http://127.16.122.129:45503/ using document root <none> and password file <none>
I20260923 23:10:06.546150 18673 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:06.546198 18673 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:06.546298 18673 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:06.547412 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "9276341c994e4b5abd7963e54f70fae1"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "afa3a753f57781929b9f26b24249e11c"
server_key_iv: "2e12f5a79e88657d1ce49ea9836898f5"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.547725 18673 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "9276341c994e4b5abd7963e54f70fae1"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "afa3a753f57781929b9f26b24249e11c"
server_key_iv: "2e12f5a79e88657d1ce49ea9836898f5"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.548971 18673 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:06.549685 18688 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.549844 18673 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:06.549919 18673 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "9276341c994e4b5abd7963e54f70fae1"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "afa3a753f57781929b9f26b24249e11c"
server_key_iv: "2e12f5a79e88657d1ce49ea9836898f5"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.549973 18673 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:06.557997 18673 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:06.559983 18673 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:06.560088 18673 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:06.560527 18673 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:06.560566 18673 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.560585 18673 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:06.560597 18673 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.566386 18673 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:33635
I20260923 23:10:06.566504 18803 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:33635 every 8 connection(s)
I20260923 23:10:06.566744 18673 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:06.570042 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.560985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:43521 (local address 127.16.122.190:36229)
0923 23:10:06.561053 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.561056 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.561273 (+ 217us) server_negotiation.cc:407] Connection header received
0923 23:10:06.561791 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.561792 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.561805 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.561878 (+ 73us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.562699 (+ 821us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.563474 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.564899 (+ 1425us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.565215 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.565291 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.567748 (+ 2457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.567749 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.567749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.567756 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.568836 (+ 1080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.569174 (+ 338us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.569175 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.569176 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.569234 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.569449 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.569450 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.569450 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.569573 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.569627 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.569874 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.569957 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:10:06.570190 18697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.560957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.129:43521)
0923 23:10:06.561136 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.561177 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.561687 (+ 510us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.561964 (+ 277us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.561968 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.562310 (+ 342us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.562596 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.562602 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.563566 (+ 964us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.563567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.564799 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.564802 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.565303 (+ 501us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.565304 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.565457 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.565999 (+ 542us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.566009 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.567655 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.568906 (+ 1251us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.568912 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.568915 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.569112 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.569286 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.569287 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.569287 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.569396 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.569626 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.569632 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.569762 (+ 130us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.569847 (+ 85us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.569987 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":101,"threads_started":1}
I20260923 23:10:06.570677 18804 heartbeater.cc:347] Connected to a master server at 127.16.122.190:36229
I20260923 23:10:06.570749 18804 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:06.570902 18804 heartbeater.cc:511] Master 127.16.122.190:36229 requested a full tablet report, sending...
I20260923 23:10:06.571408 18600 ts_manager.cc:194] Registered new tserver with Master: 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635)
I20260923 23:10:06.572254 18600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:43521
I20260923 23:10:06.576424 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.571790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44371 (local address 127.16.122.190:36229)
0923 23:10:06.571829 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.571831 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.571872 (+ 41us) server_negotiation.cc:407] Connection header received
0923 23:10:06.571925 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.571926 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.571934 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.571979 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.572329 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.572932 (+ 603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.573584 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.573771 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.573831 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.574500 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.574501 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.574502 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.574507 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.575559 (+ 1052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.575853 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.575854 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.575854 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.575914 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.576057 (+ 143us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.576057 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.576057 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.576161 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.576216 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.576306 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.576368 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:06.576467 18810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.571706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.129:44371)
0923 23:10:06.571817 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.571827 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.571882 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.571984 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.571985 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.572154 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.572212 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.572214 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.573026 (+ 812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.573027 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.573499 (+ 472us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.573500 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.573832 (+ 332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.573832 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.573880 (+ 48us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.574199 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.574202 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.574440 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.575619 (+ 1179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.575620 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.575620 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.575777 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.575972 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.575972 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.575973 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.576015 (+ 42us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.576207 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.576208 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.576239 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.576278 (+ 39us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.576379 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":87,"thread_start_us":60,"threads_started":1}
I20260923 23:10:06.576489 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18673
I20260923 23:10:06.576567 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:06.576691 16874 external_mini_cluster.cc:1428] Setting key 85898d79df5dabb8b1b50c986863cb36
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:06.601511 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:36229
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:06.679818 18813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:06.680023 18813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:06.680047 18813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:06.681180 18813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:06.681218 18813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:06.681236 18813 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:10:06.681257 18813 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:06.681286 18813 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:06.682675 18813 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:06.682888 18813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:06.683117 18813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:06.687747 18819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:06.687757 18818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.687813 18813 server_base.cc:1034] running on GCE node
W20260923 23:10:06.687757 18821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.688136 18813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:06.688336 18813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:06.689467 18813 hybrid_clock.cc:648] HybridClock initialized: now 1790205006689454 us; error 29 us; skew 500 ppm
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:06.694691 18813 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:06.695143 18813 webserver.cc:460] Webserver started at http://127.16.122.130:41455/ using document root <none> and password file <none>
I20260923 23:10:06.695353 18813 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:06.695400 18813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:06.695524 18813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:06.696539 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "757306ab3b8e49ffab8e3f1d14caf271"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "fb14fdc2ba1559b495ff90fabf1e360d"
server_key_iv: "386bd0c2ece54728f290a66436b5507b"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.696867 18813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "757306ab3b8e49ffab8e3f1d14caf271"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "fb14fdc2ba1559b495ff90fabf1e360d"
server_key_iv: "386bd0c2ece54728f290a66436b5507b"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.698063 18813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:06.698685 18828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.698829 18813 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:06.698907 18813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "757306ab3b8e49ffab8e3f1d14caf271"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "fb14fdc2ba1559b495ff90fabf1e360d"
server_key_iv: "386bd0c2ece54728f290a66436b5507b"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.698971 18813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:06.717186 18813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:06.719347 18813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:06.719484 18813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:06.719971 18813 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:06.720016 18813 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.720043 18813 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:06.720059 18813 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:06.726521 18813 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:41431
I20260923 23:10:06.726586 18943 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:41431 every 8 connection(s)
I20260923 23:10:06.726905 18813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:06.728665 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.720414 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:53795 (local address 127.16.122.190:36229)
0923 23:10:06.720457 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.720459 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.720687 (+ 228us) server_negotiation.cc:407] Connection header received
0923 23:10:06.721188 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.721190 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.721203 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.721252 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.722048 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.722799 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.723915 (+ 1116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.724103 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.724157 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.726361 (+ 2204us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.726363 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.726363 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.726370 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.727697 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.728015 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.728016 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.728016 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.728060 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.728212 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.728213 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.728213 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.728308 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.728353 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.728552 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.728604 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:06.728852 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.720424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.130:53795)
0923 23:10:06.720568 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.720603 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.721093 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.721329 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.721334 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.721656 (+ 322us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.721942 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.721948 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.722894 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.722895 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.723834 (+ 939us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.723837 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.724181 (+ 344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.724181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.724301 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.724762 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.724772 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.726272 (+ 1500us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.727760 (+ 1488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.727763 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.727765 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.727961 (+ 196us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.728095 (+ 134us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.728096 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.728096 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.728175 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.728351 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.728354 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.728460 (+ 106us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.728537 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.728649 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":82,"threads_started":1}
I20260923 23:10:06.729256 18944 heartbeater.cc:347] Connected to a master server at 127.16.122.190:36229
I20260923 23:10:06.729331 18944 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:06.729496 18944 heartbeater.cc:511] Master 127.16.122.190:36229 requested a full tablet report, sending...
I20260923 23:10:06.729831 18601 ts_manager.cc:194] Registered new tserver with Master: 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431)
I20260923 23:10:06.730304 18601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:53795
I20260923 23:10:06.734555 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.730202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:53209 (local address 127.16.122.190:36229)
0923 23:10:06.730240 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.730241 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.730301 (+ 60us) server_negotiation.cc:407] Connection header received
0923 23:10:06.730360 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.730360 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.730369 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.730427 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.730746 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.731328 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.731952 (+ 624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.732121 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.732161 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.732818 (+ 657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.732819 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.732819 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.732825 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.733795 (+ 970us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.734051 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.734051 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.734052 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.734093 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.734198 (+ 105us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.734198 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.734198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.734314 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.734358 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.734448 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.734498 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:06.734566 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.730138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.130:53209)
0923 23:10:06.730252 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.730261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.730310 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.730429 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.730430 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.730614 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.730675 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.730677 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.731397 (+ 720us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.731398 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.731897 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.731899 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.732163 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.732164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.732214 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.732517 (+ 303us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.732520 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.732759 (+ 239us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.733856 (+ 1097us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.733856 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.733857 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.734002 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.734127 (+ 125us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.734127 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.734127 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.734167 (+ 40us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.734354 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.734355 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.734386 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.734429 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.734499 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":86,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260923 23:10:06.736181 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18813
I20260923 23:10:06.736263 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:06.736409 16874 external_mini_cluster.cc:1428] Setting key d13ed7e8903f739ebfd5bad095341c27
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:06.762179 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:36229
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:06.842921 18953 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:06.843138 18953 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:06.843161 18953 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:06.844255 18953 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:06.844288 18953 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:06.844307 18953 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260923 23:10:06.844323 18953 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:06.844338 18953 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:06.845700 18953 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:36229
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.18953
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:06.845880 18953 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:06.846055 18953 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:06.850718 18961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.850764 18953 server_base.cc:1034] running on GCE node
W20260923 23:10:06.850768 18959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:06.850872 18958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:06.851073 18953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:06.851269 18953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:06.852388 18953 hybrid_clock.cc:648] HybridClock initialized: now 1790205006852379 us; error 26 us; skew 500 ppm
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:06.854553 18953 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:06.854930 18953 webserver.cc:460] Webserver started at http://127.16.122.131:46243/ using document root <none> and password file <none>
I20260923 23:10:06.855154 18953 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:06.855217 18953 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:06.855353 18953 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:06.856353 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "1530770c85814035a3faef7d1ba13ee8"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "67a13de078434d285b09cf56a23e68a6"
server_key_iv: "42dc169983935dddfbf94e42e01057bd"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.856657 18953 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "1530770c85814035a3faef7d1ba13ee8"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "67a13de078434d285b09cf56a23e68a6"
server_key_iv: "42dc169983935dddfbf94e42e01057bd"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.857836 18953 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:06.858436 18968 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.858567 18953 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:06.858639 18953 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "1530770c85814035a3faef7d1ba13ee8"
format_stamp: "Formatted at 2026-09-23 23:10:06 on dist-test-slave-95hh"
server_key: "67a13de078434d285b09cf56a23e68a6"
server_key_iv: "42dc169983935dddfbf94e42e01057bd"
server_key_version: "encryptionkey@0"
I20260923 23:10:06.858702 18953 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:06.875290 18953 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:06.877013 18953 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:06.877133 18953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:06.877619 18953 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:06.877660 18953 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.877681 18953 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:06.877692 18953 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:06.883971 18953 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:42181
I20260923 23:10:06.884042 19083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:42181 every 8 connection(s)
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:06.884328 18953 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:06.887297 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.878025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:42403 (local address 127.16.122.190:36229)
0923 23:10:06.878083 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.878086 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.878282 (+ 196us) server_negotiation.cc:407] Connection header received
0923 23:10:06.878740 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.878741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.878754 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.878826 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.879604 (+ 778us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.880347 (+ 743us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.881810 (+ 1463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.882112 (+ 302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.882193 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.884832 (+ 2639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.884838 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.884839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.884846 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.886042 (+ 1196us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.886400 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.886401 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.886402 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.886451 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.886663 (+ 212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.886664 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.886664 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.886775 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.886830 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.887103 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.887200 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:06.887310 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 18953
I20260923 23:10:06.887387 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:06.887441 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.877996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.131:42403)
0923 23:10:06.878151 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.878191 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.878653 (+ 462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.878898 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.878903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.879216 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.879501 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.879509 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.880450 (+ 941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.880452 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.881717 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.881721 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.882202 (+ 481us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.882203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.882356 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.882904 (+ 548us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.882914 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.884710 (+ 1796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.886118 (+ 1408us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.886124 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.886126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.886336 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.886504 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.886504 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.886505 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.886610 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.886829 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.886835 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.886974 (+ 139us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.887077 (+ 103us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.887231 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260923 23:10:06.887545 16874 external_mini_cluster.cc:1428] Setting key 4d8b17ca526967027123e57c8814428c
I20260923 23:10:06.887988 19084 heartbeater.cc:347] Connected to a master server at 127.16.122.190:36229
I20260923 23:10:06.888068 19084 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:06.888237 19084 heartbeater.cc:511] Master 127.16.122.190:36229 requested a full tablet report, sending...
I20260923 23:10:06.888540 18601 ts_manager.cc:194] Registered new tserver with Master: 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181)
I20260923 23:10:06.888984 18601 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:42403
I20260923 23:10:06.893931 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.889323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:42757 (local address 127.16.122.190:36229)
0923 23:10:06.889355 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.889357 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.889440 (+ 83us) server_negotiation.cc:407] Connection header received
0923 23:10:06.889501 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.889501 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.889509 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.889551 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.889866 (+ 315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.890315 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.890927 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.891135 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.891179 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.891848 (+ 669us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.891849 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.891850 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.891854 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.892944 (+ 1090us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.893325 (+ 381us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.893326 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.893326 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.893379 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.893533 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.893534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.893534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.893654 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.893698 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.893820 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.893874 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260923 23:10:06.893941 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.889277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.16.122.131:42757)
0923 23:10:06.889390 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.889400 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.889452 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.889559 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.889560 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.889744 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:06.889799 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.889801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.890386 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.890387 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.890874 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.890876 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.891183 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.891184 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.891238 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.891547 (+ 309us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:06.891550 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:06.891788 (+ 238us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:06.893037 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.893038 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.893039 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.893259 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.893426 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:06.893427 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:06.893427 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:06.893484 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:06.893701 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:06.893702 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:06.893739 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.893801 (+ 62us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.893878 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":58,"threads_started":1}
I20260923 23:10:06.899269 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:10:06 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:06.919453 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.913402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43804 (local address 127.16.122.190:36229)
0923 23:10:06.913451 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.913452 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.913459 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:06.913484 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.913485 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.913494 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.913546 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:06.914047 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.914532 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.915268 (+ 736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.915455 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.915502 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.917566 (+ 2064us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:06.917567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:06.917568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:06.917575 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:06.918607 (+ 1032us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.918881 (+ 274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.918881 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.918882 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.918924 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:06.919076 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:06.919077 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:06.919077 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:06.919172 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:06.919239 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.919327 (+ 88us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.919390 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:10:06.920862 18601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:06.922326 18601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:06.928534 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924577 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50352 (local address 127.16.122.129:33635)
0923 23:10:06.924860 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.924865 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.924875 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:06.924913 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.924915 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.924930 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.925131 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:06.925187 (+ 56us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.926020 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.927421 (+ 1401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.928132 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.928209 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.928284 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.928408 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.928443 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":236,"thread_start_us":83,"threads_started":1}
I20260923 23:10:06.928529 19103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.0.0.1:50342)
0923 23:10:06.924586 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.924596 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.924649 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.925006 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.925007 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.925010 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:06.925063 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.925065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.926379 (+ 1314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.926380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.927447 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.927449 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.928265 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.928267 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.928334 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.928338 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.928364 (+ 26us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.928443 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":1,"thread_start_us":87,"threads_started":1}
I20260923 23:10:06.928623 19101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:33635 (local address 127.0.0.1:50352)
0923 23:10:06.924514 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.924555 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.924615 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.925023 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.925023 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.925024 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:06.925063 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.925065 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.926092 (+ 1027us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.926093 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.927345 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.927348 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.928214 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.928215 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.928289 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.928304 (+ 15us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.928379 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.928445 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":136,"mutex_wait_us":22,"thread_start_us":39,"threads_started":1}
I20260923 23:10:06.928965 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924503 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50342 (local address 127.16.122.131:42181)
0923 23:10:06.924837 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.924840 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.924848 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:06.924880 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.924882 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.924897 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.924970 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:06.925131 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.926288 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.927513 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.928179 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.928749 (+ 570us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.928813 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.928836 (+ 23us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.928874 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":306,"thread_start_us":68,"threads_started":1}
I20260923 23:10:06.929260 19102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41431 (local address 127.0.0.1:38314)
0923 23:10:06.924715 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:06.924722 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:06.924752 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:06.924976 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:06.924979 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:06.924982 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:06.925042 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.925043 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.927514 (+ 2471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.927514 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.928153 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:06.928154 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.929036 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.929037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:06.929132 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.929136 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:06.929165 (+ 29us) client_negotiation.cc:240] Negotiation successful
0923 23:10:06.929201 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":344,"mutex_wait_us":2,"thread_start_us":92,"threads_started":1}
I20260923 23:10:06.929574 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:06.924658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38314 (local address 127.16.122.130:41431)
0923 23:10:06.924792 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:06.924797 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:06.924807 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:06.924845 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:06.924848 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:06.924864 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:06.925107 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:06.925161 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.926296 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.928235 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:06.928952 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:06.929358 (+ 406us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:06.929415 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:06.929439 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 23:10:06.929476 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":76,"threads_started":1}
I20260923 23:10:06.929564 19018 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5176549822425ebe347a708ba0c398 (DEFAULT_TABLE table=test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:06.929597 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5176549822425ebe347a708ba0c398 (DEFAULT_TABLE table=test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:06.929937 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5176549822425ebe347a708ba0c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:06.929939 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5176549822425ebe347a708ba0c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:06.930359 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 3e5176549822425ebe347a708ba0c398 (DEFAULT_TABLE table=test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:06.930652 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 3e5176549822425ebe347a708ba0c398. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:06.932576 19107 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Bootstrap starting.
I20260923 23:10:06.933390 19107 tablet_bootstrap.cc:654] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:06.933422 19108 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Bootstrap starting.
I20260923 23:10:06.934006 19108 tablet_bootstrap.cc:654] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:06.933987 19107 log.cc:824] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:06.934116 19109 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Bootstrap starting.
I20260923 23:10:06.934312 19108 log.cc:824] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:06.934645 19107 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: No bootstrap required, opened a new log
I20260923 23:10:06.934648 19109 tablet_bootstrap.cc:654] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:06.934700 19107 ts_tablet_manager.cc:1397] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:06.934927 19109 log.cc:824] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:06.935021 19108 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: No bootstrap required, opened a new log
I20260923 23:10:06.935108 19108 ts_tablet_manager.cc:1397] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:06.935515 19109 tablet_bootstrap.cc:492] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: No bootstrap required, opened a new log
I20260923 23:10:06.935571 19109 ts_tablet_manager.cc:1397] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:06.936615 19107 raft_consensus.cc:348] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.936795 19107 raft_consensus.cc:374] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:06.936831 19107 raft_consensus.cc:729] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Initialized, Role: FOLLOWER
I20260923 23:10:06.936834 19109 raft_consensus.cc:348] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.936944 19107 consensus_queue.cc:260] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.936980 19109 raft_consensus.cc:374] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:06.936923 19108 raft_consensus.cc:348] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.937016 19109 raft_consensus.cc:729] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Initialized, Role: FOLLOWER
I20260923 23:10:06.937055 19108 raft_consensus.cc:374] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:06.937094 19108 raft_consensus.cc:729] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Initialized, Role: FOLLOWER
I20260923 23:10:06.937141 19109 consensus_queue.cc:260] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.937182 19107 ts_tablet_manager.cc:1428] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:10:06.937294 19084 heartbeater.cc:503] Master 127.16.122.190:36229 was elected leader, sending a full tablet report...
I20260923 23:10:06.937279 19108 consensus_queue.cc:260] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:06.937381 19109 ts_tablet_manager.cc:1428] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:06.937500 19108 ts_tablet_manager.cc:1428] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:10:06.937549 18944 heartbeater.cc:503] Master 127.16.122.190:36229 was elected leader, sending a full tablet report...
I20260923 23:10:06.937600 18804 heartbeater.cc:503] Master 127.16.122.190:36229 was elected leader, sending a full tablet report...
W20260923 23:10:06.977571 18945 tablet.cc:2369] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:07.001475 19114 raft_consensus.cc:482] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:07.001610 19114 raft_consensus.cc:504] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:07.001935 19114 leader_election.cc:288] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181), 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635)
I20260923 23:10:07.004571 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.002264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:45853 (local address 127.16.122.129:33635)
0923 23:10:07.002302 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.002304 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.002310 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:10:07.002333 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.002334 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.002341 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.002391 (+ 50us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:07.002528 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.003056 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.003939 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.004277 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.004336 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.004344 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.004477 (+ 133us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.004509 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:07.004601 18837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.002102 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:33635 (local address 127.16.122.130:45853)
0923 23:10:07.002156 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.002167 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.002212 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.002407 (+ 195us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.002409 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.002411 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:07.002472 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.002474 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.003133 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.003134 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.003855 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.003857 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.004344 (+ 487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.004344 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.004417 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.004420 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.004463 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.004503 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:10:07.004622 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.002181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:59573 (local address 127.16.122.131:42181)
0923 23:10:07.002231 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.002233 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.002418 (+ 185us) server_negotiation.cc:407] Connection header received
0923 23:10:07.002460 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.002461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.002469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.002510 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:07.002637 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.003160 (+ 523us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.003988 (+ 828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.004331 (+ 343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.004391 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.004399 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.004525 (+ 126us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.004557 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:07.004700 19116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.002106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.16.122.130:59573)
0923 23:10:07.002358 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.002365 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.002409 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.002512 (+ 103us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.002513 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.002514 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:07.002588 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.002589 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.003226 (+ 637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.003227 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.003915 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.003916 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.004404 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.004404 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.004468 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.004471 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.004515 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.004552 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":239,"mutex_wait_us":92,"thread_start_us":107,"threads_started":1}
I20260923 23:10:07.005007 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5176549822425ebe347a708ba0c398" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1" is_pre_election: true
I20260923 23:10:07.005007 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5176549822425ebe347a708ba0c398" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8" is_pre_election: true
I20260923 23:10:07.005156 19038 raft_consensus.cc:2454] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 0.
I20260923 23:10:07.005156 18758 raft_consensus.cc:2454] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 0.
I20260923 23:10:07.005364 18831 leader_election.cc:302] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757306ab3b8e49ffab8e3f1d14caf271, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:07.005492 19114 raft_consensus.cc:2781] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:07.005546 19114 raft_consensus.cc:482] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:07.005574 19114 raft_consensus.cc:3037] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.006474 19114 raft_consensus.cc:504] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:07.006582 19114 leader_election.cc:288] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 election: Requested vote from peers 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181), 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635)
I20260923 23:10:07.006724 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5176549822425ebe347a708ba0c398" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8"
I20260923 23:10:07.006752 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "3e5176549822425ebe347a708ba0c398" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1"
I20260923 23:10:07.006803 19038 raft_consensus.cc:3037] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.006826 18758 raft_consensus.cc:3037] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.007550 19038 raft_consensus.cc:2454] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 1.
I20260923 23:10:07.007570 18758 raft_consensus.cc:2454] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 1.
I20260923 23:10:07.007704 18831 leader_election.cc:302] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757306ab3b8e49ffab8e3f1d14caf271, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:07.007823 19114 raft_consensus.cc:2781] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:07.007973 19114 raft_consensus.cc:686] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 LEADER]: Becoming Leader. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Running, Role: LEADER
I20260923 23:10:07.008057 19114 consensus_queue.cc:237] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } }
I20260923 23:10:07.008798 18599 catalog_manager.cc:5495] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "757306ab3b8e49ffab8e3f1d14caf271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } health_report { overall_health: UNKNOWN } } }
Sep 23 23:10:07 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205006, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:07.045897 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.039656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36229 (local address 127.0.0.1:43810)
0923 23:10:07.039818 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.039828 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.039895 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.040024 (+ 129us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.040025 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.040345 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:07.040391 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.040394 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.041014 (+ 620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.041015 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.041526 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.041527 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.041831 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.041832 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.041876 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.042363 (+ 487us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:07.042367 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:07.043652 (+ 1285us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:07.044982 (+ 1330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:07.044989 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:07.044991 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:07.045163 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:07.045320 (+ 157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:07.045320 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:07.045321 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:07.045373 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:07.045562 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:07.045568 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:07.045612 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.045670 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.045776 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":90,"threads_started":1}
I20260923 23:10:07.045905 18650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.039724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43810 (local address 127.16.122.190:36229)
0923 23:10:07.039791 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.039793 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.039888 (+ 95us) server_negotiation.cc:407] Connection header received
0923 23:10:07.039951 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.039952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.039962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.040008 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:07.040471 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.040947 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.041598 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.041776 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.041828 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.043735 (+ 1907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:07.043736 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:07.043736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:07.043742 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:07.044914 (+ 1172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:07.045220 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:07.045221 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:07.045221 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:07.045269 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:07.045419 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:07.045419 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:07.045420 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:07.045513 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:07.045565 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.045696 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.045829 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":40}
I20260923 23:10:07.047777 18599 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:43810:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:10:07.047956 18599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:07.049191 18878 tablet_service.cc:1467] Processing CreateTablet for tablet b1e69c666f2e4333805b53fb3cd4c7e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dae1e76ff54d5c9dde71ef790e2bd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:07.049214 19018 tablet_service.cc:1467] Processing CreateTablet for tablet b1e69c666f2e4333805b53fb3cd4c7e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dae1e76ff54d5c9dde71ef790e2bd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:07.049214 18738 tablet_service.cc:1467] Processing CreateTablet for tablet b1e69c666f2e4333805b53fb3cd4c7e7 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=70dae1e76ff54d5c9dde71ef790e2bd6]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:07.049337 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e69c666f2e4333805b53fb3cd4c7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.049364 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e69c666f2e4333805b53fb3cd4c7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.049364 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b1e69c666f2e4333805b53fb3cd4c7e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.050477 19109 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: Bootstrap starting.
I20260923 23:10:07.050792 19107 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: Bootstrap starting.
I20260923 23:10:07.050813 19108 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: Bootstrap starting.
I20260923 23:10:07.050882 19109 tablet_bootstrap.cc:654] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.051290 19107 tablet_bootstrap.cc:654] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.051476 19108 tablet_bootstrap.cc:654] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.051581 19109 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: No bootstrap required, opened a new log
I20260923 23:10:07.051620 19109 ts_tablet_manager.cc:1397] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:07.051754 19109 raft_consensus.cc:348] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.051846 19109 raft_consensus.cc:374] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.051865 19109 raft_consensus.cc:729] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.051870 19107 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: No bootstrap required, opened a new log
I20260923 23:10:07.051908 19107 ts_tablet_manager.cc:1397] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:07.051904 19109 consensus_queue.cc:260] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.051997 19109 ts_tablet_manager.cc:1428] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:07.052039 19108 tablet_bootstrap.cc:492] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: No bootstrap required, opened a new log
I20260923 23:10:07.052037 19107 raft_consensus.cc:348] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.052074 19108 ts_tablet_manager.cc:1397] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:07.052099 19107 raft_consensus.cc:374] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.052119 19107 raft_consensus.cc:729] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.052121 19114 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.052160 19107 consensus_queue.cc:260] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.052198 19114 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.052235 19107 ts_tablet_manager.cc:1428] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:07.052243 19108 raft_consensus.cc:348] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.052310 19108 raft_consensus.cc:374] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.052328 19108 raft_consensus.cc:729] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.052366 19108 consensus_queue.cc:260] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.052446 19108 ts_tablet_manager.cc:1428] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:07.052475 19113 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.052547 19113 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.052579 19115 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.052645 19115 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 23:10:07.067526 18805 tablet.cc:2369] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:07.134990 19085 tablet.cc:2369] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:07.137881 19113 raft_consensus.cc:482] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:07.137952 19113 raft_consensus.cc:504] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.138278 19113 leader_election.cc:288] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431)
I20260923 23:10:07.141034 19105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.138542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:35079 (local address 127.16.122.129:33635)
0923 23:10:07.138594 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.138596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.138636 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:10:07.138665 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.138666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.138676 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.138728 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:07.138908 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.139443 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140370 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.140718 (+ 348us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140774 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.140783 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.140933 (+ 150us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.140967 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:10:07.141039 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.138529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:44897 (local address 127.16.122.130:41431)
0923 23:10:07.138575 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.138578 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.138737 (+ 159us) server_negotiation.cc:407] Connection header received
0923 23:10:07.138780 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.138781 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.138789 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.138838 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:07.138969 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.139457 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140373 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.140731 (+ 358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140793 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.140802 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.140941 (+ 139us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.140980 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:10:07.141037 18977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.138463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:33635 (local address 127.16.122.131:35079)
0923 23:10:07.138518 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.138530 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.138575 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.138752 (+ 177us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.138753 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.138768 (+ 15us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:07.138842 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.138844 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.139521 (+ 677us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.139522 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.140285 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.140287 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.140790 (+ 503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140791 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.140870 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.140873 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.140915 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.140954 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37,"mutex_wait_us":3}
I20260923 23:10:07.141055 19128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.138465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41431 (local address 127.16.122.131:44897)
0923 23:10:07.138654 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.138665 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.138727 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.138845 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.138845 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.138847 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:07.138908 (+ 61us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.138909 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.139530 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.139531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.140283 (+ 752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.140285 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.140823 (+ 538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.140824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.140876 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.140879 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.140926 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.140967 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":170,"mutex_wait_us":17,"thread_start_us":105,"threads_started":1}
I20260923 23:10:07.141314 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e69c666f2e4333805b53fb3cd4c7e7" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1" is_pre_election: true
I20260923 23:10:07.141364 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e69c666f2e4333805b53fb3cd4c7e7" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271" is_pre_election: true
I20260923 23:10:07.141413 18758 raft_consensus.cc:2454] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 0.
I20260923 23:10:07.141462 18898 raft_consensus.cc:2454] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 0.
I20260923 23:10:07.141587 18971 leader_election.cc:302] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1530770c85814035a3faef7d1ba13ee8, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:07.141736 19113 raft_consensus.cc:2781] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:07.141769 19113 raft_consensus.cc:482] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:07.141788 19113 raft_consensus.cc:3037] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.142509 19113 raft_consensus.cc:504] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.142629 19113 leader_election.cc:288] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 election: Requested vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431)
I20260923 23:10:07.142755 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e69c666f2e4333805b53fb3cd4c7e7" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271"
I20260923 23:10:07.142815 18898 raft_consensus.cc:3037] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.142802 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b1e69c666f2e4333805b53fb3cd4c7e7" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1"
I20260923 23:10:07.142879 18758 raft_consensus.cc:3037] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.143497 18898 raft_consensus.cc:2454] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 1.
I20260923 23:10:07.143498 18758 raft_consensus.cc:2454] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 1.
I20260923 23:10:07.143666 18969 leader_election.cc:302] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1530770c85814035a3faef7d1ba13ee8, 757306ab3b8e49ffab8e3f1d14caf271; no voters:
I20260923 23:10:07.143752 19113 raft_consensus.cc:2781] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:07.143901 19113 raft_consensus.cc:686] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Running, Role: LEADER
I20260923 23:10:07.143985 19113 consensus_queue.cc:237] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.144420 19129 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: TxnStatusTablet state changed. Reason: New leader 1530770c85814035a3faef7d1ba13ee8. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.144487 19129 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:07.144685 19131 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:10:07.144801 19131 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:10:07.144734 18599 catalog_manager.cc:5495] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } health_report { overall_health: HEALTHY } } }
I20260923 23:10:07.169502 19124 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.167221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.0.0.1:50344)
0923 23:10:07.167284 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.167295 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.167340 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.167500 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.167501 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.167502 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:07.167552 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.167555 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.168140 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.168142 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.168703 (+ 561us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.168705 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.169029 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.169030 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.169095 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.169111 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.169339 (+ 228us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.169347 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.169392 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.169434 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:10:07.169502 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.167287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50344 (local address 127.16.122.131:42181)
0923 23:10:07.167350 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.167352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.167361 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:07.167415 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.167416 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.167435 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.167491 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.167607 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.168074 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.168773 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.168971 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.169025 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.169185 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.169288 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.169363 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.169399 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.169417 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:07.171492 18898 raft_consensus.cc:1261] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Refusing update from remote peer 1530770c85814035a3faef7d1ba13ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.171541 18758 raft_consensus.cc:1261] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Refusing update from remote peer 1530770c85814035a3faef7d1ba13ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.171844 19129 consensus_queue.cc:1035] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:07.171993 19113 consensus_queue.cc:1035] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:07.173190 19114 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: TxnStatusTablet state changed. Reason: New leader 1530770c85814035a3faef7d1ba13ee8. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.173278 19114 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.173358 19115 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: TxnStatusTablet state changed. Reason: New leader 1530770c85814035a3faef7d1ba13ee8. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.173432 19115 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.174403 19113 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.174479 19113 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:07.174638 19114 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.174714 19114 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 757306ab3b8e49ffab8e3f1d14caf271: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.174861 19134 mvcc.cc:205] Tried to move back new op lower bound from 7332679709371781120 to 7332679709262282752. Current Snapshot: MvccSnapshot[applied={T|T < 7332679709371781120}]
I20260923 23:10:07.174949 19136 mvcc.cc:205] Tried to move back new op lower bound from 7332679709371781120 to 7332679709262282752. Current Snapshot: MvccSnapshot[applied={T|T < 7332679709371781120}]
I20260923 23:10:07.174999 19113 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.175094 19113 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 1530770c85814035a3faef7d1ba13ee8: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:07.175486 19115 tablet_replica.cc:439] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } } }
I20260923 23:10:07.175568 19115 tablet_replica.cc:442] T b1e69c666f2e4333805b53fb3cd4c7e7 P 9276341c994e4b5abd7963e54f70fae1: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:07.175853 19135 mvcc.cc:205] Tried to move back new op lower bound from 7332679709371781120 to 7332679709262282752. Current Snapshot: MvccSnapshot[applied={T|T < 7332679709371781120 or (T in {7332679709371781120})}]
I20260923 23:10:07.183424 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.180823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38324 (local address 127.16.122.130:41431)
0923 23:10:07.180869 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.180871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.180880 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:07.180932 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.180933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.180947 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.181002 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.181136 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.181596 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.182683 (+ 1087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.182871 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.182917 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.183130 (+ 213us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.183234 (+ 104us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.183290 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.183345 (+ 55us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.183364 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:07.188017 18950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.185830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.16.122.130:43871)
0923 23:10:07.185899 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.185910 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.185952 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.186066 (+ 114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.186066 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.186067 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:07.186123 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.186125 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.186694 (+ 569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.186695 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.187305 (+ 610us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.187307 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.187617 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.187618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.187680 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.187697 (+ 17us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.187867 (+ 170us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.187868 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.187915 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.187955 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:10:07.188014 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.185912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:43871 (local address 127.16.122.131:42181)
0923 23:10:07.185969 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.185972 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.185989 (+ 17us) server_negotiation.cc:407] Connection header received
0923 23:10:07.186011 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.186012 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.186021 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.186059 (+ 38us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.186177 (+ 118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.186630 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.187378 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.187563 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.187615 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.187789 (+ 174us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.187818 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.187865 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.187918 (+ 53us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.187938 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:07.190479 18758 raft_consensus.cc:1261] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Refusing update from remote peer 757306ab3b8e49ffab8e3f1d14caf271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.190613 19038 raft_consensus.cc:1261] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Refusing update from remote peer 757306ab3b8e49ffab8e3f1d14caf271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.190901 19114 consensus_queue.cc:1035] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:07.191054 19117 consensus_queue.cc:1035] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:07.198792 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.196425 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41431 (local address 127.16.122.131:46157)
0923 23:10:07.196483 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.196495 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.196537 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.196743 (+ 206us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.196744 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.196745 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:07.196789 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.196790 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.197449 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.197450 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.198005 (+ 555us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.198007 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.198363 (+ 356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.198364 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.198434 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.198450 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.198610 (+ 160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.198630 (+ 20us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.198666 (+ 36us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.198715 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 23:10:07.198979 19106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.196585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:46157 (local address 127.16.122.130:41431)
0923 23:10:07.196619 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.196621 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.196628 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:07.196646 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.196647 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.196655 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.196701 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.196839 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.197380 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.198074 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.198284 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.198352 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.198531 (+ 179us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.198558 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.198595 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.198895 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.198916 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:07.205577 18599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:07.205675 18599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:07.206357 18599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Sending DeleteTablet for 3 replicas of tablet 3e5176549822425ebe347a708ba0c398
I20260923 23:10:07.206645 19018 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5176549822425ebe347a708ba0c398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:07 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50342
I20260923 23:10:07.206813 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5176549822425ebe347a708ba0c398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:07 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38314
I20260923 23:10:07.206816 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 3e5176549822425ebe347a708ba0c398 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:07 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260923 23:10:07.206820 18599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:07.206971 18599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:07.208143 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca9815584420aed005a293277354 (DEFAULT_TABLE table=test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:07.208194 19018 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca9815584420aed005a293277354 (DEFAULT_TABLE table=test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:07.208243 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 5fbeca9815584420aed005a293277354 (DEFAULT_TABLE table=test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:07.208276 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca9815584420aed005a293277354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.208305 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca9815584420aed005a293277354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.208323 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 5fbeca9815584420aed005a293277354. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:07.209579 19109 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Bootstrap starting.
I20260923 23:10:07.209630 19107 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Bootstrap starting.
I20260923 23:10:07.209928 19109 tablet_bootstrap.cc:654] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.209929 19107 tablet_bootstrap.cc:654] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.210177 19108 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Bootstrap starting.
I20260923 23:10:07.210534 19108 tablet_bootstrap.cc:654] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:07.210983 19107 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: No bootstrap required, opened a new log
I20260923 23:10:07.211052 19107 ts_tablet_manager.cc:1397] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:07.211154 19108 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: No bootstrap required, opened a new log
I20260923 23:10:07.211203 19108 ts_tablet_manager.cc:1397] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:07.211301 19107 raft_consensus.cc:348] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.211333 19108 raft_consensus.cc:348] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.211380 19107 raft_consensus.cc:374] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.211388 19108 raft_consensus.cc:374] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.211401 19107 raft_consensus.cc:729] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.211406 19108 raft_consensus.cc:729] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.211445 19108 consensus_queue.cc:260] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.211449 19107 consensus_queue.cc:260] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.211526 19108 ts_tablet_manager.cc:1428] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:07.211536 19107 ts_tablet_manager.cc:1428] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:07.215193 19154 tablet_replica.cc:330] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: stopping tablet replica
I20260923 23:10:07.215327 19154 raft_consensus.cc:2229] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:07.215440 19154 raft_consensus.cc:2258] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:07.215826 19154 ts_tablet_manager.cc:1905] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:07.217408 19154 ts_tablet_manager.cc:1918] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:07.217470 19154 log.cc:1196] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/3e5176549822425ebe347a708ba0c398
I20260923 23:10:07.217716 19154 ts_tablet_manager.cc:1939] T 3e5176549822425ebe347a708ba0c398 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting consensus metadata
I20260923 23:10:07.218080 18585 catalog_manager.cc:4841] TS 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431): tablet 3e5176549822425ebe347a708ba0c398 (table test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]) successfully deleted
I20260923 23:10:07.219244 19153 tablet_replica.cc:330] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: stopping tablet replica
I20260923 23:10:07.219376 19153 raft_consensus.cc:2229] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:07.219398 19109 tablet_bootstrap.cc:492] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: No bootstrap required, opened a new log
I20260923 23:10:07.219447 19109 ts_tablet_manager.cc:1397] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.000s
I20260923 23:10:07.219460 19153 raft_consensus.cc:2258] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:07.219626 19152 tablet_replica.cc:330] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: stopping tablet replica
I20260923 23:10:07.219595 19109 raft_consensus.cc:348] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.219673 19109 raft_consensus.cc:374] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:07.219702 19109 raft_consensus.cc:729] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Initialized, Role: FOLLOWER
I20260923 23:10:07.219756 19152 raft_consensus.cc:2229] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:07.219749 19109 consensus_queue.cc:260] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.219817 19152 raft_consensus.cc:2258] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:07.219849 19109 ts_tablet_manager.cc:1428] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:07.219971 19153 ts_tablet_manager.cc:1905] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:07.220458 19152 ts_tablet_manager.cc:1905] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:07.221541 19153 ts_tablet_manager.cc:1918] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:07.221616 19153 log.cc:1196] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/3e5176549822425ebe347a708ba0c398
I20260923 23:10:07.221678 19152 ts_tablet_manager.cc:1918] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:07.221748 19152 log.cc:1196] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/3e5176549822425ebe347a708ba0c398
I20260923 23:10:07.221858 19153 ts_tablet_manager.cc:1939] T 3e5176549822425ebe347a708ba0c398 P 9276341c994e4b5abd7963e54f70fae1: Deleting consensus metadata
I20260923 23:10:07.222019 19152 ts_tablet_manager.cc:1939] T 3e5176549822425ebe347a708ba0c398 P 1530770c85814035a3faef7d1ba13ee8: Deleting consensus metadata
I20260923 23:10:07.222282 18587 catalog_manager.cc:4841] TS 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635): tablet 3e5176549822425ebe347a708ba0c398 (table test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]) successfully deleted
I20260923 23:10:07.222380 18588 catalog_manager.cc:4841] TS 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181): tablet 3e5176549822425ebe347a708ba0c398 (table test-table [id=e31945b82b2941bda3cf2b72ddb4c3e8]) successfully deleted
I20260923 23:10:07.354709 19150 raft_consensus.cc:482] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:07.354810 19150 raft_consensus.cc:504] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.354948 19150 leader_election.cc:288] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431)
I20260923 23:10:07.355190 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca9815584420aed005a293277354" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271" is_pre_election: true
I20260923 23:10:07.355191 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca9815584420aed005a293277354" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1" is_pre_election: true
I20260923 23:10:07.355293 18898 raft_consensus.cc:2454] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 0.
I20260923 23:10:07.355295 18758 raft_consensus.cc:2454] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 0.
I20260923 23:10:07.355445 18969 leader_election.cc:302] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1530770c85814035a3faef7d1ba13ee8, 757306ab3b8e49ffab8e3f1d14caf271; no voters:
I20260923 23:10:07.355554 19150 raft_consensus.cc:2781] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:07.355605 19150 raft_consensus.cc:482] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:07.355625 19150 raft_consensus.cc:3037] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.356383 19150 raft_consensus.cc:504] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.356501 19150 leader_election.cc:288] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 election: Requested vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431)
I20260923 23:10:07.356632 18758 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca9815584420aed005a293277354" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1"
I20260923 23:10:07.356654 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "5fbeca9815584420aed005a293277354" candidate_uuid: "1530770c85814035a3faef7d1ba13ee8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271"
I20260923 23:10:07.356715 18898 raft_consensus.cc:3037] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.356719 18758 raft_consensus.cc:3037] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:07.357245 18898 raft_consensus.cc:2454] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 1.
I20260923 23:10:07.357322 18758 raft_consensus.cc:2454] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1530770c85814035a3faef7d1ba13ee8 in term 1.
I20260923 23:10:07.357430 18969 leader_election.cc:302] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1530770c85814035a3faef7d1ba13ee8, 757306ab3b8e49ffab8e3f1d14caf271; no voters:
I20260923 23:10:07.357524 19150 raft_consensus.cc:2781] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:07.357578 19150 raft_consensus.cc:686] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 1 LEADER]: Becoming Leader. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Running, Role: LEADER
I20260923 23:10:07.357627 19150 consensus_queue.cc:237] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:07.358197 18599 catalog_manager.cc:5495] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "1530770c85814035a3faef7d1ba13ee8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } health_report { overall_health: HEALTHY } } }
I20260923 23:10:07.395490 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.393323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50348 (local address 127.16.122.131:42181)
0923 23:10:07.393360 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.393362 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.393394 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:10:07.393453 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.393454 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.393468 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.393512 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.393631 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.394108 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.394803 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.395002 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.395095 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.395243 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.395277 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.395319 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.395383 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.395404 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:07.399096 19104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.397310 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:37817 (local address 127.16.122.131:42181)
0923 23:10:07.397352 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:07.397353 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:07.397360 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:07.397391 (+ 31us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:07.397392 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:07.397399 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:07.397438 (+ 39us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:07.397510 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.397936 (+ 426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.398580 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.398756 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.398814 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.398895 (+ 81us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.398916 (+ 21us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.398947 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:07.398987 (+ 40us) server_negotiation.cc:299] Negotiation successful
0923 23:10:07.398999 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:07.399155 19090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:07.397251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.16.122.131:37817)
0923 23:10:07.397299 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:07.397310 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:07.397349 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:07.397441 (+ 92us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:07.397441 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:07.397442 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:07.397475 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.397475 (+ 0us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.397984 (+ 509us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.397984 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.398506 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:07.398507 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:07.398817 (+ 310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:07.398818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:07.398850 (+ 32us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:07.398854 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:07.398955 (+ 101us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:07.398956 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:07.398984 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 23:10:07.399011 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 23:10:07.400981 18758 raft_consensus.cc:1261] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Refusing update from remote peer 1530770c85814035a3faef7d1ba13ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.400992 18898 raft_consensus.cc:1261] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Refusing update from remote peer 1530770c85814035a3faef7d1ba13ee8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:07.401165 19129 consensus_queue.cc:1035] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:07.401271 19150 consensus_queue.cc:1035] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:10:07.405143 19134 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 23:10:08.922369 18601 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:08.922472 18601 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43804:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:08.923290 18601 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Sending DeleteTablet for 3 replicas of tablet 5fbeca9815584420aed005a293277354
I20260923 23:10:08.923542 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fbeca9815584420aed005a293277354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38314
I20260923 23:10:08.923620 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fbeca9815584420aed005a293277354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260923 23:10:08.923779 19182 tablet_replica.cc:330] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: stopping tablet replica
I20260923 23:10:08.923770 19018 tablet_service.cc:1514] Processing DeleteTablet for tablet 5fbeca9815584420aed005a293277354 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50342
I20260923 23:10:08.923823 19182 raft_consensus.cc:2229] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:08.923856 19182 raft_consensus.cc:2258] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.924018 19183 tablet_replica.cc:330] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: stopping tablet replica
I20260923 23:10:08.924062 19182 ts_tablet_manager.cc:1905] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.924077 19183 raft_consensus.cc:2229] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:08.924155 19183 raft_consensus.cc:2258] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.924368 19183 ts_tablet_manager.cc:1905] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.925446 19182 ts_tablet_manager.cc:1918] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:10:08.925485 19182 log.cc:1196] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/5fbeca9815584420aed005a293277354
I20260923 23:10:08.925629 19182 ts_tablet_manager.cc:1939] T 5fbeca9815584420aed005a293277354 P 9276341c994e4b5abd7963e54f70fae1: Deleting consensus metadata
I20260923 23:10:08.925902 19183 ts_tablet_manager.cc:1918] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:10:08.925902 18587 catalog_manager.cc:4841] TS 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635): tablet 5fbeca9815584420aed005a293277354 (table test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]) successfully deleted
I20260923 23:10:08.925951 19183 log.cc:1196] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/5fbeca9815584420aed005a293277354
I20260923 23:10:08.926167 19183 ts_tablet_manager.cc:1939] T 5fbeca9815584420aed005a293277354 P 1530770c85814035a3faef7d1ba13ee8: Deleting consensus metadata
I20260923 23:10:08.926364 19181 tablet_replica.cc:330] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: stopping tablet replica
I20260923 23:10:08.926429 19181 raft_consensus.cc:2229] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:08.926469 19181 raft_consensus.cc:2258] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.926530 18588 catalog_manager.cc:4841] TS 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181): tablet 5fbeca9815584420aed005a293277354 (table test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]) successfully deleted
I20260923 23:10:08.926684 19181 ts_tablet_manager.cc:1905] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.928208 19181 ts_tablet_manager.cc:1918] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:10:08.928265 19181 log.cc:1196] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/5fbeca9815584420aed005a293277354
I20260923 23:10:08.928444 19181 ts_tablet_manager.cc:1939] T 5fbeca9815584420aed005a293277354 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting consensus metadata
I20260923 23:10:08.928740 18585 catalog_manager.cc:4841] TS 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431): tablet 5fbeca9815584420aed005a293277354 (table test-table [id=f8cb645ddebf4cd2ad74fb1ba53d4bb4]) successfully deleted
Sep 23 23:10:08 dist-test-slave-95hh krb5kdc[18556](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:10:08 dist-test-slave-95hh krb5kdc[18556](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205008, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:08.946973 19191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:08.940089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43822 (local address 127.16.122.190:36229)
0923 23:10:08.940215 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:08.940217 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:08.940226 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:08.940272 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:08.940273 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:08.940283 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:08.940346 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:08.940859 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:08.941385 (+ 526us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:08.942143 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:08.942350 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:08.942400 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:08.944634 (+ 2234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:08.944635 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:08.944635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:08.944642 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:08.945979 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:08.946284 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:08.946285 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:08.946285 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:08.946343 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:08.946558 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:08.946558 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:08.946558 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:08.946661 (+ 103us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:08.946729 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:08.946830 (+ 101us) server_negotiation.cc:299] Negotiation successful
0923 23:10:08.946890 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100,"thread_start_us":70,"threads_started":1}
I20260923 23:10:08.948401 18601 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:08.948555 18601 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:08.949875 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbe96569af740b4a68f19bb603933e6 (DEFAULT_TABLE table=test-table [id=711b602ef6304be8b5da9851680844bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.949874 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbe96569af740b4a68f19bb603933e6 (DEFAULT_TABLE table=test-table [id=711b602ef6304be8b5da9851680844bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.949894 19018 tablet_service.cc:1467] Processing CreateTablet for tablet 4cbe96569af740b4a68f19bb603933e6 (DEFAULT_TABLE table=test-table [id=711b602ef6304be8b5da9851680844bd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.950007 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbe96569af740b4a68f19bb603933e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.950009 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbe96569af740b4a68f19bb603933e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.950008 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4cbe96569af740b4a68f19bb603933e6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.950989 19192 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Bootstrap starting.
I20260923 23:10:08.951361 19192 tablet_bootstrap.cc:654] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.951380 19194 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Bootstrap starting.
I20260923 23:10:08.951716 19194 tablet_bootstrap.cc:654] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.951998 19193 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Bootstrap starting.
I20260923 23:10:08.952025 19192 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: No bootstrap required, opened a new log
I20260923 23:10:08.952104 19192 ts_tablet_manager.cc:1397] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:08.952255 19194 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: No bootstrap required, opened a new log
I20260923 23:10:08.952253 19192 raft_consensus.cc:348] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.952297 19194 ts_tablet_manager.cc:1397] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:08.952316 19192 raft_consensus.cc:374] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.952335 19192 raft_consensus.cc:729] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.952389 19193 tablet_bootstrap.cc:654] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.952404 19192 consensus_queue.cc:260] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.952404 19194 raft_consensus.cc:348] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.952468 19194 raft_consensus.cc:374] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.952486 19194 raft_consensus.cc:729] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.952533 19194 consensus_queue.cc:260] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.952551 19192 ts_tablet_manager.cc:1428] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:08.952699 19194 ts_tablet_manager.cc:1428] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:08.953038 19193 tablet_bootstrap.cc:492] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: No bootstrap required, opened a new log
I20260923 23:10:08.953088 19193 ts_tablet_manager.cc:1397] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:08.953226 19193 raft_consensus.cc:348] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.953300 19193 raft_consensus.cc:374] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.953320 19193 raft_consensus.cc:729] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.953354 19193 consensus_queue.cc:260] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.953455 19193 ts_tablet_manager.cc:1428] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:08.960760 19198 raft_consensus.cc:482] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:08.960815 19198 raft_consensus.cc:504] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.960943 19198 leader_election.cc:288] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181)
I20260923 23:10:08.961073 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbe96569af740b4a68f19bb603933e6" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8" is_pre_election: true
I20260923 23:10:08.961089 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbe96569af740b4a68f19bb603933e6" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1" is_pre_election: true
I20260923 23:10:08.961150 19038 raft_consensus.cc:2454] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 0.
I20260923 23:10:08.961151 18757 raft_consensus.cc:2454] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 0.
I20260923 23:10:08.961304 18832 leader_election.cc:302] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1530770c85814035a3faef7d1ba13ee8, 757306ab3b8e49ffab8e3f1d14caf271; no voters:
I20260923 23:10:08.961407 19198 raft_consensus.cc:2781] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:08.961453 19198 raft_consensus.cc:482] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:08.961472 19198 raft_consensus.cc:3037] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:08.962067 19198 raft_consensus.cc:504] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.962184 19198 leader_election.cc:288] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 election: Requested vote from peers 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635), 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181)
I20260923 23:10:08.962303 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbe96569af740b4a68f19bb603933e6" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8"
I20260923 23:10:08.962334 18757 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4cbe96569af740b4a68f19bb603933e6" candidate_uuid: "757306ab3b8e49ffab8e3f1d14caf271" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9276341c994e4b5abd7963e54f70fae1"
I20260923 23:10:08.962383 19038 raft_consensus.cc:3037] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:08.962391 18757 raft_consensus.cc:3037] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:08.963084 19038 raft_consensus.cc:2454] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 1.
I20260923 23:10:08.963084 18757 raft_consensus.cc:2454] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 757306ab3b8e49ffab8e3f1d14caf271 in term 1.
I20260923 23:10:08.963241 18831 leader_election.cc:302] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757306ab3b8e49ffab8e3f1d14caf271, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:08.963336 19198 raft_consensus.cc:2781] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:08.963464 19198 raft_consensus.cc:686] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 LEADER]: Becoming Leader. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Running, Role: LEADER
I20260923 23:10:08.963502 19198 consensus_queue.cc:237] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.963936 18599 catalog_manager.cc:5495] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 reported cstate change: term changed from 0 to 1, leader changed from <none> to 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "757306ab3b8e49ffab8e3f1d14caf271" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:08.971362 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:08.969139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38340 (local address 127.16.122.130:41431)
0923 23:10:08.969267 (+ 128us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:08.969269 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:08.969274 (+ 5us) server_negotiation.cc:407] Connection header received
0923 23:10:08.969290 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:08.969291 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:08.969299 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:08.969352 (+ 53us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:08.969482 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:08.969957 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:08.970684 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:08.970922 (+ 238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:08.970970 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:08.971141 (+ 171us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:08.971174 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:08.971219 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:08.971266 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 23:10:08.971287 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":73,"threads_started":1}
I20260923 23:10:08.973289 19038 raft_consensus.cc:1261] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Refusing update from remote peer 757306ab3b8e49ffab8e3f1d14caf271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:08.973313 18757 raft_consensus.cc:1261] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Refusing update from remote peer 757306ab3b8e49ffab8e3f1d14caf271: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:08.973536 19198 consensus_queue.cc:1035] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:08.973632 19200 consensus_queue.cc:1035] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:08.982810 18599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:08.982904 18599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:08.983464 18599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Sending DeleteTablet for 3 replicas of tablet 4cbe96569af740b4a68f19bb603933e6
I20260923 23:10:08.983676 19018 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbe96569af740b4a68f19bb603933e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50342
I20260923 23:10:08.983665 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbe96569af740b4a68f19bb603933e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260923 23:10:08.983762 19183 tablet_replica.cc:330] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: stopping tablet replica
I20260923 23:10:08.983779 19182 tablet_replica.cc:330] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: stopping tablet replica
I20260923 23:10:08.983810 19183 raft_consensus.cc:2229] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:08.983819 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 4cbe96569af740b4a68f19bb603933e6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:08 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38314
I20260923 23:10:08.983835 19182 raft_consensus.cc:2229] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:08.983842 19183 raft_consensus.cc:2258] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.983872 19182 raft_consensus.cc:2258] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.983891 19181 tablet_replica.cc:330] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: stopping tablet replica
I20260923 23:10:08.983932 19181 raft_consensus.cc:2229] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:08.983989 19181 raft_consensus.cc:2258] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:08.984098 18599 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:08.984222 18599 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:08.984277 19183 ts_tablet_manager.cc:1905] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.984504 19181 ts_tablet_manager.cc:1905] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.985268 18738 tablet_service.cc:1467] Processing CreateTablet for tablet 7b03cb826d7f4624950fa88cafc86993 (DEFAULT_TABLE table=test-table [id=6450570e780b4ba48f25e6dd28e224f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.985288 19018 tablet_service.cc:1467] Processing CreateTablet for tablet 7b03cb826d7f4624950fa88cafc86993 (DEFAULT_TABLE table=test-table [id=6450570e780b4ba48f25e6dd28e224f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.985364 18738 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b03cb826d7f4624950fa88cafc86993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.985383 19018 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b03cb826d7f4624950fa88cafc86993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.985847 19183 ts_tablet_manager.cc:1918] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:08.985895 19183 log.cc:1196] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/4cbe96569af740b4a68f19bb603933e6
I20260923 23:10:08.985977 19181 ts_tablet_manager.cc:1918] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:08.986025 19181 log.cc:1196] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/4cbe96569af740b4a68f19bb603933e6
I20260923 23:10:08.986093 19183 ts_tablet_manager.cc:1939] T 4cbe96569af740b4a68f19bb603933e6 P 1530770c85814035a3faef7d1ba13ee8: Deleting consensus metadata
I20260923 23:10:08.986212 19181 ts_tablet_manager.cc:1939] T 4cbe96569af740b4a68f19bb603933e6 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting consensus metadata
I20260923 23:10:08.986344 19182 ts_tablet_manager.cc:1905] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:08.986522 19192 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1: Bootstrap starting.
I20260923 23:10:08.986528 19193 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Bootstrap starting.
I20260923 23:10:08.986753 19193 tablet_bootstrap.cc:654] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.987092 19192 tablet_bootstrap.cc:654] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.987147 18585 catalog_manager.cc:4841] TS 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431): tablet 4cbe96569af740b4a68f19bb603933e6 (table test-table [id=711b602ef6304be8b5da9851680844bd]) successfully deleted
I20260923 23:10:08.987215 18878 tablet_service.cc:1467] Processing CreateTablet for tablet 7b03cb826d7f4624950fa88cafc86993 (DEFAULT_TABLE table=test-table [id=6450570e780b4ba48f25e6dd28e224f1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:08.987294 18878 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 7b03cb826d7f4624950fa88cafc86993. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:08.987352 19193 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: No bootstrap required, opened a new log
I20260923 23:10:08.987390 19193 ts_tablet_manager.cc:1397] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:08.987481 19193 raft_consensus.cc:348] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.987560 19193 raft_consensus.cc:374] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.987578 19193 raft_consensus.cc:729] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1530770c85814035a3faef7d1ba13ee8, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.987588 19182 ts_tablet_manager.cc:1918] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:08.987630 19182 log.cc:1196] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/4cbe96569af740b4a68f19bb603933e6
I20260923 23:10:08.987617 19193 consensus_queue.cc:260] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.987697 19193 ts_tablet_manager.cc:1428] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:08.987818 19182 ts_tablet_manager.cc:1939] T 4cbe96569af740b4a68f19bb603933e6 P 9276341c994e4b5abd7963e54f70fae1: Deleting consensus metadata
I20260923 23:10:08.988101 18587 catalog_manager.cc:4841] TS 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635): tablet 4cbe96569af740b4a68f19bb603933e6 (table test-table [id=711b602ef6304be8b5da9851680844bd]) successfully deleted
I20260923 23:10:08.988405 19194 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Bootstrap starting.
I20260923 23:10:08.988425 18588 catalog_manager.cc:4841] TS 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181): tablet 4cbe96569af740b4a68f19bb603933e6 (table test-table [id=711b602ef6304be8b5da9851680844bd]) successfully deleted
I20260923 23:10:08.988682 19194 tablet_bootstrap.cc:654] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:08.988735 19192 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1: No bootstrap required, opened a new log
I20260923 23:10:08.988778 19192 ts_tablet_manager.cc:1397] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:10:08.988897 19192 raft_consensus.cc:348] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.988958 19192 raft_consensus.cc:374] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.988976 19192 raft_consensus.cc:729] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.989017 19192 consensus_queue.cc:260] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.989120 19192 ts_tablet_manager.cc:1428] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:08.989188 19194 tablet_bootstrap.cc:492] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: No bootstrap required, opened a new log
I20260923 23:10:08.989233 19194 ts_tablet_manager.cc:1397] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:08.989326 19194 raft_consensus.cc:348] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.989406 19194 raft_consensus.cc:374] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:08.989425 19194 raft_consensus.cc:729] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 757306ab3b8e49ffab8e3f1d14caf271, State: Initialized, Role: FOLLOWER
I20260923 23:10:08.989461 19194 consensus_queue.cc:260] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:08.989538 19194 ts_tablet_manager.cc:1428] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:09.191485 19197 raft_consensus.cc:482] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:09.191586 19197 raft_consensus.cc:504] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:09.191931 19197 leader_election.cc:288] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431), 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181)
I20260923 23:10:09.196014 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.192216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:45501 (local address 127.16.122.131:42181)
0923 23:10:09.192310 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:09.192312 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:09.192323 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:09.192386 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:09.192387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:09.192399 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:09.192479 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:09.192672 (+ 193us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.193454 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195018 (+ 1564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.195641 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195707 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.195716 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:09.195903 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 23:10:09.195949 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72,"thread_start_us":42,"threads_started":1}
I20260923 23:10:09.196009 19203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.192167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44939 (local address 127.16.122.130:41431)
0923 23:10:09.192217 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:09.192219 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:09.192394 (+ 175us) server_negotiation.cc:407] Connection header received
0923 23:10:09.192433 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:09.192434 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:09.192442 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:09.192493 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:09.192672 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.193452 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195018 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.195604 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195699 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.195710 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:09.195894 (+ 184us) server_negotiation.cc:299] Negotiation successful
0923 23:10:09.195935 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:09.196022 19211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.192097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.16.122.129:45501)
0923 23:10:09.192252 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:09.192261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:09.192316 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:09.192509 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:09.192509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:09.192516 (+ 7us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:09.192608 (+ 92us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.192610 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.193540 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.193540 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.194911 (+ 1371us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.194914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.195706 (+ 792us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195707 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.195811 (+ 104us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.195817 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:10:09.195883 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:10:09.195919 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":142,"mutex_wait_us":90,"thread_start_us":68,"threads_started":1}
I20260923 23:10:09.196027 19212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.192101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41431 (local address 127.16.122.129:44939)
0923 23:10:09.192331 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:09.192339 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:09.192390 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:09.192509 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:09.192509 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:09.192512 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:09.192613 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.192614 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.193523 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.193524 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.194912 (+ 1388us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.194914 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.195696 (+ 782us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.195697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.195811 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.195816 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:09.195874 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:10:09.195919 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":213,"mutex_wait_us":54,"thread_start_us":118,"threads_started":1}
I20260923 23:10:09.196244 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b03cb826d7f4624950fa88cafc86993" candidate_uuid: "9276341c994e4b5abd7963e54f70fae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271" is_pre_election: true
I20260923 23:10:09.196261 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b03cb826d7f4624950fa88cafc86993" candidate_uuid: "9276341c994e4b5abd7963e54f70fae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8" is_pre_election: true
I20260923 23:10:09.196324 18898 raft_consensus.cc:2454] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9276341c994e4b5abd7963e54f70fae1 in term 0.
I20260923 23:10:09.196350 19038 raft_consensus.cc:2454] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9276341c994e4b5abd7963e54f70fae1 in term 0.
I20260923 23:10:09.196503 18689 leader_election.cc:302] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757306ab3b8e49ffab8e3f1d14caf271, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:09.196650 19197 raft_consensus.cc:2781] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:09.196688 19197 raft_consensus.cc:482] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:09.196702 19197 raft_consensus.cc:3037] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:09.197343 19197 raft_consensus.cc:504] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:09.197458 19197 leader_election.cc:288] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [CANDIDATE]: Term 1 election: Requested vote from peers 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431), 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181)
I20260923 23:10:09.197611 18898 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b03cb826d7f4624950fa88cafc86993" candidate_uuid: "9276341c994e4b5abd7963e54f70fae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "757306ab3b8e49ffab8e3f1d14caf271"
I20260923 23:10:09.197643 19038 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "7b03cb826d7f4624950fa88cafc86993" candidate_uuid: "9276341c994e4b5abd7963e54f70fae1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1530770c85814035a3faef7d1ba13ee8"
I20260923 23:10:09.197698 19038 raft_consensus.cc:3037] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:09.197698 18898 raft_consensus.cc:3037] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:09.198269 19038 raft_consensus.cc:2454] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9276341c994e4b5abd7963e54f70fae1 in term 1.
I20260923 23:10:09.198268 18898 raft_consensus.cc:2454] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9276341c994e4b5abd7963e54f70fae1 in term 1.
I20260923 23:10:09.198438 18689 leader_election.cc:302] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 757306ab3b8e49ffab8e3f1d14caf271, 9276341c994e4b5abd7963e54f70fae1; no voters:
I20260923 23:10:09.198529 19197 raft_consensus.cc:2781] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:09.198662 19197 raft_consensus.cc:686] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [term 1 LEADER]: Becoming Leader. State: Replica: 9276341c994e4b5abd7963e54f70fae1, State: Running, Role: LEADER
I20260923 23:10:09.198716 19197 consensus_queue.cc:237] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } }
I20260923 23:10:09.199407 18601 catalog_manager.cc:5495] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129). New cstate: current_term: 1 leader_uuid: "9276341c994e4b5abd7963e54f70fae1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9276341c994e4b5abd7963e54f70fae1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 33635 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:09.217222 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.214908 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50362 (local address 127.16.122.129:33635)
0923 23:10:09.215075 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:09.215077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:09.215083 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:10:09.215121 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:09.215122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:09.215140 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:09.215200 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:09.215304 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.215790 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.216423 (+ 633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.216628 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.216676 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.216779 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:09.216881 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:09.216931 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:09.217066 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:10:09.217090 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":92,"threads_started":1}
I20260923 23:10:09.221522 19213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.219578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:49417 (local address 127.16.122.131:42181)
0923 23:10:09.219630 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:09.219632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:09.219680 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:10:09.219737 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:09.219737 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:09.219744 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:09.219784 (+ 40us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:09.219893 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.220329 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.221026 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.221199 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.221240 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.221367 (+ 127us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:09.221390 (+ 23us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:09.221421 (+ 31us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:09.221451 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 23:10:09.221467 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:09.221558 19219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.219418 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:42181 (local address 127.16.122.129:49417)
0923 23:10:09.219611 (+ 193us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:09.219623 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:09.219694 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:09.219792 (+ 98us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:09.219793 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:09.219794 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:09.219843 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.219844 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.220395 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.220395 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.220972 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.220974 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.221261 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.221262 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.221303 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.221315 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:09.221420 (+ 105us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:09.221421 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:09.221448 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 23:10:09.221480 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":86,"threads_started":1}
I20260923 23:10:09.223352 19038 raft_consensus.cc:1261] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Refusing update from remote peer 9276341c994e4b5abd7963e54f70fae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:09.223352 18898 raft_consensus.cc:1261] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Refusing update from remote peer 9276341c994e4b5abd7963e54f70fae1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:09.223593 19214 consensus_queue.cc:1035] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1530770c85814035a3faef7d1ba13ee8" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 42181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:09.223722 19197 consensus_queue.cc:1035] T 7b03cb826d7f4624950fa88cafc86993 P 9276341c994e4b5abd7963e54f70fae1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "757306ab3b8e49ffab8e3f1d14caf271" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41431 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:09.229877 19216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.227718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:34369 (local address 127.16.122.129:33635)
0923 23:10:09.227759 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:09.227761 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:09.227913 (+ 152us) server_negotiation.cc:407] Connection header received
0923 23:10:09.227936 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:09.227937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:09.227947 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:09.227993 (+ 46us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:09.228079 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.228520 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.229249 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.229485 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.229531 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.229689 (+ 158us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:09.229722 (+ 33us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:09.229759 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:09.229809 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 23:10:09.229830 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:09.229928 19224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:09.227685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:33635 (local address 127.16.122.131:34369)
0923 23:10:09.227825 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:09.227835 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:09.227890 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:09.227995 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:09.227996 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:09.227999 (+ 3us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:09.228040 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.228041 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.228573 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.228574 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.229167 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:09.229170 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:09.229542 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:09.229543 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:09.229604 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:09.229609 (+ 5us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:09.229770 (+ 161us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:09.229771 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:09.229800 (+ 29us) client_negotiation.cc:240] Negotiation successful
0923 23:10:09.229845 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":88,"threads_started":1}
W20260923 23:10:09.230196 19165 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260923 23:10:10.746829 18599 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:10.746927 18599 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:10.747660 18599 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 5f0539a70f3c47e09a93157b7999bc8b: Sending DeleteTablet for 3 replicas of tablet 7b03cb826d7f4624950fa88cafc86993
I20260923 23:10:10.747880 18738 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b03cb826d7f4624950fa88cafc86993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:10 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50352
I20260923 23:10:10.747973 19018 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b03cb826d7f4624950fa88cafc86993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:10 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50342
I20260923 23:10:10.748055 18878 tablet_service.cc:1514] Processing DeleteTablet for tablet 7b03cb826d7f4624950fa88cafc86993 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:10 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38314
I20260923 23:10:10.748095 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18673
I20260923 23:10:10.748688 19249 tablet_replica.cc:330] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: stopping tablet replica
I20260923 23:10:10.748771 19249 raft_consensus.cc:2229] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:10.748814 19249 raft_consensus.cc:2258] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:10.748996 19250 tablet_replica.cc:330] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: stopping tablet replica
I20260923 23:10:10.749056 19250 raft_consensus.cc:2229] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:10.749065 19249 ts_tablet_manager.cc:1905] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:10.749092 19250 raft_consensus.cc:2258] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:10.749313 19250 ts_tablet_manager.cc:1905] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:10.750725 19250 ts_tablet_manager.cc:1918] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:10:10.750746 19249 ts_tablet_manager.cc:1918] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260923 23:10:10.750777 19250 log.cc:1196] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/7b03cb826d7f4624950fa88cafc86993
I20260923 23:10:10.750795 19249 log.cc:1196] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/7b03cb826d7f4624950fa88cafc86993
I20260923 23:10:10.751013 19250 ts_tablet_manager.cc:1939] T 7b03cb826d7f4624950fa88cafc86993 P 757306ab3b8e49ffab8e3f1d14caf271: Deleting consensus metadata
I20260923 23:10:10.751021 19249 ts_tablet_manager.cc:1939] T 7b03cb826d7f4624950fa88cafc86993 P 1530770c85814035a3faef7d1ba13ee8: Deleting consensus metadata
I20260923 23:10:10.751346 18585 catalog_manager.cc:4841] TS 757306ab3b8e49ffab8e3f1d14caf271 (127.16.122.130:41431): tablet 7b03cb826d7f4624950fa88cafc86993 (table test-table [id=6450570e780b4ba48f25e6dd28e224f1]) successfully deleted
I20260923 23:10:10.751384 18588 catalog_manager.cc:4841] TS 1530770c85814035a3faef7d1ba13ee8 (127.16.122.131:42181): tablet 7b03cb826d7f4624950fa88cafc86993 (table test-table [id=6450570e780b4ba48f25e6dd28e224f1]) successfully deleted
W20260923 23:10:10.754956 18587 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:33635 (error 108)
I20260923 23:10:10.755172 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18813
I20260923 23:10:10.756289 19252 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:10.755310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:33635 (local address 127.0.0.1:57446)
0923 23:10:10.756190 (+ 880us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:10.756247 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:33635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":55,"threads_started":1}
W20260923 23:10:10.756420 18587 catalog_manager.cc:4584] TS 9276341c994e4b5abd7963e54f70fae1 (127.16.122.129:33635): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 7b03cb826d7f4624950fa88cafc86993: Network error: Client connection negotiation failed: client connection to 127.16.122.129:33635: connect: Connection refused (error 111)
I20260923 23:10:10.760568 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18953
I20260923 23:10:10.766129 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 18569
2026-09-23T23:10:10Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (5554 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:10 dist-test-slave-95hh krb5kdc[19255](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:10 dist-test-slave-95hh krb5kdc[19255](info): set up 2 sockets
Sep 23 23:10:10 dist-test-slave-95hh krb5kdc[19255](info): commencing operation
krb5kdc: starting...
W20260923 23:10:11.712322 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:11 dist-test-slave-95hh krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:11.821363 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34511
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34511
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:11.899245 19268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:11.899471 19268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:11.899503 19268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:11.900691 19268 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:11.900738 19268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:11.900755 19268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:11.900780 19268 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:11.900796 19268 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:11.902259 19268 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34511
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34511
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19268
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:11.902503 19268 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:11.902706 19268 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:11.907357 19273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:11.907369 19276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:11.907415 19268 server_base.cc:1034] running on GCE node
W20260923 23:10:11.907395 19274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:11.907826 19268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:11.908105 19268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:11.909242 19268 hybrid_clock.cc:648] HybridClock initialized: now 1790205011909232 us; error 28 us; skew 500 ppm
Sep 23 23:10:11 dist-test-slave-95hh krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205011, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:11.911595 19268 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:11.911973 19268 webserver.cc:460] Webserver started at http://127.16.122.190:40993/ using document root <none> and password file <none>
I20260923 23:10:11.912156 19268 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:11.912194 19268 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:11.912285 19268 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:11.913244 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "b1b6684627e742e496a052d07f317222"
format_stamp: "Formatted at 2026-09-23 23:10:11 on dist-test-slave-95hh"
server_key: "7284e3a24fdbf0ee6e71419b2c68b342"
server_key_iv: "3475f4aabed46e60ef458ae184e950b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:11.913525 19268 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "b1b6684627e742e496a052d07f317222"
format_stamp: "Formatted at 2026-09-23 23:10:11 on dist-test-slave-95hh"
server_key: "7284e3a24fdbf0ee6e71419b2c68b342"
server_key_iv: "3475f4aabed46e60ef458ae184e950b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:11.914690 19268 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:10:11.915431 19283 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:11.915588 19268 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:11.915663 19268 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "b1b6684627e742e496a052d07f317222"
format_stamp: "Formatted at 2026-09-23 23:10:11 on dist-test-slave-95hh"
server_key: "7284e3a24fdbf0ee6e71419b2c68b342"
server_key_iv: "3475f4aabed46e60ef458ae184e950b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:11.915724 19268 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:11.930282 19268 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:11.932431 19268 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:11.932560 19268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:11.936302 19268 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34511
I20260923 23:10:11.936331 19335 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34511 every 8 connection(s)
I20260923 23:10:11.936656 19268 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:11.937445 19336 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:11.939704 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222: Bootstrap starting.
I20260923 23:10:11.940249 19336 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:11.940512 19336 log.cc:824] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:11.941136 19336 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222: No bootstrap required, opened a new log
I20260923 23:10:11.942658 19336 raft_consensus.cc:348] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } }
I20260923 23:10:11.942790 19336 raft_consensus.cc:374] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:11.942811 19336 raft_consensus.cc:729] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1b6684627e742e496a052d07f317222, State: Initialized, Role: FOLLOWER
I20260923 23:10:11.942939 19336 consensus_queue.cc:260] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } }
I20260923 23:10:11.943006 19336 raft_consensus.cc:388] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:11.943053 19336 raft_consensus.cc:482] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:11.943095 19336 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:11.943578 19336 raft_consensus.cc:504] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } }
I20260923 23:10:11.943663 19336 leader_election.cc:302] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b1b6684627e742e496a052d07f317222; no voters:
I20260923 23:10:11.943816 19336 leader_election.cc:288] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:11.943861 19339 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:11.944034 19336 sys_catalog.cc:564] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:11.944183 19339 raft_consensus.cc:686] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [term 1 LEADER]: Becoming Leader. State: Replica: b1b6684627e742e496a052d07f317222, State: Running, Role: LEADER
I20260923 23:10:11.944274 19339 consensus_queue.cc:237] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } }
I20260923 23:10:11.944615 19341 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b1b6684627e742e496a052d07f317222. Latest consensus state: current_term: 1 leader_uuid: "b1b6684627e742e496a052d07f317222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } } }
I20260923 23:10:11.944725 19341 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:11.944617 19340 sys_catalog.cc:455] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b1b6684627e742e496a052d07f317222" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b1b6684627e742e496a052d07f317222" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34511 } } }
I20260923 23:10:11.944913 19340 sys_catalog.cc:458] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:11.944903 19348 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:11.945317 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19268
I20260923 23:10:11.945395 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:11.945482 19348 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:11.945550 16874 external_mini_cluster.cc:1428] Setting key 58aec98865f1dac4445b6bb106429968
I20260923 23:10:11.947467 19348 catalog_manager.cc:1347] Generated new cluster ID: f7bc2dcff1404bd1b586ad903aef94d7
I20260923 23:10:11.947518 19348 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:11 dist-test-slave-95hh krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205011, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:11.958483 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:11.945948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46588 (local address 127.16.122.190:34511)
0923 23:10:11.946282 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:11.946291 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:11.946324 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:10:11.946435 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:11.946441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:11.946508 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:11.946645 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:11.951245 (+ 4600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:11.952214 (+ 969us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:11.952948 (+ 734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:11.953382 (+ 434us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:11.953477 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:11.955656 (+ 2179us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:11.955660 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:11.955660 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:11.955673 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:11.957132 (+ 1459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:11.957442 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:11.957445 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:11.957445 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:11.957524 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:11.957685 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:11.957686 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:11.957686 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:11.957912 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:11.958063 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:11.958130 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:10:11.958260 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":82,"threads_started":1}
I20260923 23:10:11.979549 19348 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:11.980299 19348 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:11.984330 19348 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b1b6684627e742e496a052d07f317222: Generated new TSK 0
I20260923 23:10:11.984504 19348 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:12.010196 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:12.088420 19362 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:12.088610 19362 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:12.088631 19362 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:12.089759 19362 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:12.089798 19362 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:12.089826 19362 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:12.091235 19362 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19362
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:12.091454 19362 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:12.091655 19362 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:12.096195 19370 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.096202 19368 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.096197 19367 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:12.096223 19362 server_base.cc:1034] running on GCE node
I20260923 23:10:12.096592 19362 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:12.096789 19362 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:12.097925 19362 hybrid_clock.cc:648] HybridClock initialized: now 1790205012097919 us; error 33 us; skew 500 ppm
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:12.100226 19362 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:12.100620 19362 webserver.cc:460] Webserver started at http://127.16.122.129:38137/ using document root <none> and password file <none>
I20260923 23:10:12.100812 19362 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:12.100858 19362 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:12.100979 19362 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:12.101966 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "b53da9bda60d4bb88de0b232693148d1"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "009fcd430da59a910cf15bf6bb290dbd"
server_key_iv: "f37ad74f4615b112ca6e81b805fe58d0"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.102283 19362 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "b53da9bda60d4bb88de0b232693148d1"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "009fcd430da59a910cf15bf6bb290dbd"
server_key_iv: "f37ad74f4615b112ca6e81b805fe58d0"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.103533 19362 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:12.104210 19377 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.104390 19362 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:12.104478 19362 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "b53da9bda60d4bb88de0b232693148d1"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "009fcd430da59a910cf15bf6bb290dbd"
server_key_iv: "f37ad74f4615b112ca6e81b805fe58d0"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.104539 19362 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:12.123951 19362 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:12.125735 19362 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:12.125869 19362 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:12.126080 19362 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:12.126372 19362 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:12.126405 19362 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.126432 19362 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:12.126448 19362 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.131872 19362 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:43029
I20260923 23:10:12.131940 19490 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:43029 every 8 connection(s)
I20260923 23:10:12.132232 19362 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:12.134681 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19362
I20260923 23:10:12.134754 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:12.134913 16874 external_mini_cluster.cc:1428] Setting key 2ab5e769278fb0bb26db71dc91032797
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:12.141937 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.132918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:56091 (local address 127.16.122.190:34511)
0923 23:10:12.132964 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:12.132966 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:12.133148 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:10:12.133614 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:12.133615 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:12.133625 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:12.133675 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:12.134603 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.135264 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.136305 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.136591 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.136666 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.139060 (+ 2394us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:12.139062 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:12.139062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:12.139073 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:12.140443 (+ 1370us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.140886 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.140886 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.140887 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.140933 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.141243 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.141244 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.141245 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.141376 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:12.141446 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:12.141793 (+ 347us) server_negotiation.cc:299] Negotiation successful
0923 23:10:12.141871 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:12.142151 19493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.132889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34511 (local address 127.16.122.129:56091)
0923 23:10:12.133031 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:12.133066 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:12.133538 (+ 472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:12.133731 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:12.133735 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:12.134293 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:12.134514 (+ 221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.134520 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.135348 (+ 828us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.135348 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.136214 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.136217 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.136674 (+ 457us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.136676 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.136821 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.137345 (+ 524us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:12.137356 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:12.138934 (+ 1578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:12.140521 (+ 1587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.140529 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.140531 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.140817 (+ 286us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.140985 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.140986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.140986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.141123 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.141449 (+ 326us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:12.141455 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:12.141619 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 23:10:12.141739 (+ 120us) client_negotiation.cc:240] Negotiation successful
0923 23:10:12.141913 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:10:12.142678 19491 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34511
I20260923 23:10:12.142779 19491 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:12.142980 19491 heartbeater.cc:511] Master 127.16.122.190:34511 requested a full tablet report, sending...
I20260923 23:10:12.143486 19300 ts_manager.cc:194] Registered new tserver with Master: b53da9bda60d4bb88de0b232693148d1 (127.16.122.129:43029)
I20260923 23:10:12.144191 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:56091
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:12.158181 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:12.235544 19496 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:12.235750 19496 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:12.235772 19496 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:12.236933 19496 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:12.236972 19496 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:12.236997 19496 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:12.238394 19496 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19496
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:12.238623 19496 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:12.238834 19496 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:12.243559 19501 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.243621 19504 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.243553 19502 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:12.243574 19496 server_base.cc:1034] running on GCE node
I20260923 23:10:12.243954 19496 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:12.244243 19496 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:12.245410 19496 hybrid_clock.cc:648] HybridClock initialized: now 1790205012245395 us; error 31 us; skew 500 ppm
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:12.247850 19496 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:12.248255 19496 webserver.cc:460] Webserver started at http://127.16.122.130:38945/ using document root <none> and password file <none>
I20260923 23:10:12.248453 19496 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:12.248495 19496 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:12.248618 19496 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:12.249662 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "51e59fb312d44ba684a9fa299c0d4983"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "f8f3b674abf4fe2904d2eb8a613881dc"
server_key_iv: "539a0faedb329543c8be6898a97ffec8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.249987 19496 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "51e59fb312d44ba684a9fa299c0d4983"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "f8f3b674abf4fe2904d2eb8a613881dc"
server_key_iv: "539a0faedb329543c8be6898a97ffec8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.251191 19496 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:12.251828 19511 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.251966 19496 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:12.252038 19496 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "51e59fb312d44ba684a9fa299c0d4983"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "f8f3b674abf4fe2904d2eb8a613881dc"
server_key_iv: "539a0faedb329543c8be6898a97ffec8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.252097 19496 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:12.266185 19496 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:12.268005 19496 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:12.268110 19496 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:12.268281 19496 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:12.268523 19496 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:12.268550 19496 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.268571 19496 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:12.268585 19496 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.273794 19496 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:34551
I20260923 23:10:12.273921 19624 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:34551 every 8 connection(s)
I20260923 23:10:12.274148 19496 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:12.282373 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.274815 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:43789 (local address 127.16.122.190:34511)
0923 23:10:12.274873 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:12.274875 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:12.275075 (+ 200us) server_negotiation.cc:407] Connection header received
0923 23:10:12.275547 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:12.275547 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:12.275555 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:12.275611 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:12.276491 (+ 880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.276990 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.277899 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.278071 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.278115 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.280119 (+ 2004us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:12.280121 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:12.280122 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:12.280129 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:12.281177 (+ 1048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.281533 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.281534 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.281534 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.281586 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.281790 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.281791 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.281791 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.281896 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:12.281951 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:12.282196 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 23:10:12.282278 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:12.282531 19627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.274791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34511 (local address 127.16.122.130:43789)
0923 23:10:12.274945 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:12.274985 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:12.275471 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:12.275668 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:12.275671 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:12.276219 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:12.276417 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.276422 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.277055 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.277055 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.277838 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.277840 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.278124 (+ 284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.278124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.278217 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.278681 (+ 464us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:12.278689 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:12.280013 (+ 1324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:12.281256 (+ 1243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.281261 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.281264 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.281467 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.281638 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.281638 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.281639 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.281736 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.281948 (+ 212us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:12.281955 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:12.282083 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 23:10:12.282170 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:10:12.282326 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":71,"threads_started":1}
I20260923 23:10:12.282737 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19496
I20260923 23:10:12.282836 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:12.283010 16874 external_mini_cluster.cc:1428] Setting key d2d99c5e81ded4032ef8c1a04b12abf6
I20260923 23:10:12.282998 19625 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34511
I20260923 23:10:12.283128 19625 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:12.283285 19625 heartbeater.cc:511] Master 127.16.122.190:34511 requested a full tablet report, sending...
I20260923 23:10:12.283625 19300 ts_manager.cc:194] Registered new tserver with Master: 51e59fb312d44ba684a9fa299c0d4983 (127.16.122.130:34551)
I20260923 23:10:12.283972 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:43789
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:12.305562 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34217
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:12.383586 19630 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:12.383790 19630 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:12.383810 19630 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:12.384899 19630 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:12.384932 19630 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:12.384956 19630 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:12.386284 19630 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34217
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34511
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19630
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:12.386479 19630 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:12.386654 19630 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:12.391382 19638 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.391479 19636 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:12.391382 19635 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:12.391593 19630 server_base.cc:1034] running on GCE node
I20260923 23:10:12.391788 19630 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:12.392037 19630 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:12.393178 19630 hybrid_clock.cc:648] HybridClock initialized: now 1790205012393165 us; error 28 us; skew 500 ppm
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:12.395560 19630 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:12.396026 19630 webserver.cc:460] Webserver started at http://127.16.122.131:34323/ using document root <none> and password file <none>
I20260923 23:10:12.396237 19630 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:12.396284 19630 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:12.396404 19630 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:12.397396 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "7d2e5a59b0434f72b263dcf6d795be27"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "c7410973a5e71bd2aa4e788f5f0db682"
server_key_iv: "8b9ed132c64d91a53674c6147571a1a8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.397722 19630 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "7d2e5a59b0434f72b263dcf6d795be27"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "c7410973a5e71bd2aa4e788f5f0db682"
server_key_iv: "8b9ed132c64d91a53674c6147571a1a8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.398972 19630 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:12.399694 19645 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.399837 19630 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:12.399927 19630 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "7d2e5a59b0434f72b263dcf6d795be27"
format_stamp: "Formatted at 2026-09-23 23:10:12 on dist-test-slave-95hh"
server_key: "c7410973a5e71bd2aa4e788f5f0db682"
server_key_iv: "8b9ed132c64d91a53674c6147571a1a8"
server_key_version: "encryptionkey@0"
I20260923 23:10:12.399991 19630 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:12.414023 19630 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:12.416015 19630 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:12.416146 19630 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:12.416356 19630 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:12.416776 19630 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:12.416833 19630 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.416872 19630 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:12.416889 19630 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:12.422395 19630 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:34533
I20260923 23:10:12.422444 19758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:34533 every 8 connection(s)
I20260923 23:10:12.422735 19630 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19255](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205012, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:12.430069 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19630
I20260923 23:10:12.430171 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:12.430435 16874 external_mini_cluster.cc:1428] Setting key ed6b23598fcd31f8806452a575279ca8
I20260923 23:10:12.431679 19358 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.423447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:34739 (local address 127.16.122.190:34511)
0923 23:10:12.423502 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:12.423510 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:12.423692 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:10:12.424193 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:12.424195 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:12.424206 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:12.424253 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:12.425189 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.425689 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.426717 (+ 1028us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.426913 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.426980 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.428987 (+ 2007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:12.428989 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:12.428989 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:12.428996 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:12.430237 (+ 1241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.430712 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.430714 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.430714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.430783 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:12.431013 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:12.431014 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:12.431014 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:12.431166 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:12.431240 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:12.431500 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 23:10:12.431585 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:12.431865 19761 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:12.423409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34511 (local address 127.16.122.131:34739)
0923 23:10:12.423581 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:12.423615 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:12.424100 (+ 485us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:12.424339 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:12.424343 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:12.424903 (+ 560us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:12.425096 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.425101 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.425780 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.425781 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.426632 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:12.426634 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:12.426988 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:12.426989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:12.427135 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:12.427564 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:12.427572 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:12.428901 (+ 1329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:12.430302 (+ 1401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.430310 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.430312 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.430633 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.430842 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:12.430843 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:12.430843 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:12.430953 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:12.431228 (+ 275us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:12.431234 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:12.431374 (+ 140us) client_negotiation.cc:769] Sending connection context
0923 23:10:12.431472 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 23:10:12.431633 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":82,"threads_started":1}
I20260923 23:10:12.432292 19759 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34511
I20260923 23:10:12.432384 19759 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:12.432555 19759 heartbeater.cc:511] Master 127.16.122.190:34511 requested a full tablet report, sending...
I20260923 23:10:12.432894 19300 ts_manager.cc:194] Registered new tserver with Master: 7d2e5a59b0434f72b263dcf6d795be27 (127.16.122.131:34533)
I20260923 23:10:12.433238 19300 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:34739
I20260923 23:10:12.441979 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:10:12.452159 19358 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:12.451332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46590 (local address 127.16.122.190:34511)
0923 23:10:12.451408 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:12.451410 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:12.451491 (+ 81us) server_negotiation.cc:407] Connection header received
0923 23:10:12.451545 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:12.451546 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:12.451555 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:12.451618 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:12.452089 (+ 471us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:46590: BlockingRecv error: recv got EOF from 127.0.0.1:46590 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 23:10:12.452970 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19362
I20260923 23:10:12.456923 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19496
I20260923 23:10:12.460549 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19630
I20260923 23:10:12.464164 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19268
2026-09-23T23:10:12Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (1697 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19771](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19771](info): set up 2 sockets
Sep 23 23:10:12 dist-test-slave-95hh krb5kdc[19771](info): commencing operation
krb5kdc: starting...
W20260923 23:10:13.409103 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:13.519825 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:44529
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:44529
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:13.598104 19784 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:13.598327 19784 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:13.598357 19784 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:13.599542 19784 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:13.599586 19784 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:13.599602 19784 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:13.599619 19784 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:13.599634 19784 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:13.601171 19784 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:44529
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:44529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19784
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:13.601420 19784 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:13.601620 19784 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:13.606204 19789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:13.606288 19790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:13.606210 19792 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:13.606238 19784 server_base.cc:1034] running on GCE node
I20260923 23:10:13.606642 19784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:13.606895 19784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:13.608031 19784 hybrid_clock.cc:648] HybridClock initialized: now 1790205013608022 us; error 41 us; skew 500 ppm
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19771](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205013, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:13.610350 19784 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:13.610785 19784 webserver.cc:460] Webserver started at http://127.16.122.190:40539/ using document root <none> and password file <none>
I20260923 23:10:13.610982 19784 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:13.611047 19784 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:13.611174 19784 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:13.612198 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "bb73aebb51dc4c1daa6bf155249dd7a2"
format_stamp: "Formatted at 2026-09-23 23:10:13 on dist-test-slave-95hh"
server_key: "fbefccf675f1baadd9fae41a1d89bab9"
server_key_iv: "2cc17133f01778092e37e802477df9bf"
server_key_version: "encryptionkey@0"
I20260923 23:10:13.612521 19784 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "bb73aebb51dc4c1daa6bf155249dd7a2"
format_stamp: "Formatted at 2026-09-23 23:10:13 on dist-test-slave-95hh"
server_key: "fbefccf675f1baadd9fae41a1d89bab9"
server_key_iv: "2cc17133f01778092e37e802477df9bf"
server_key_version: "encryptionkey@0"
I20260923 23:10:13.613732 19784 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:10:13.614403 19799 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:13.614547 19784 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:13.614627 19784 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "bb73aebb51dc4c1daa6bf155249dd7a2"
format_stamp: "Formatted at 2026-09-23 23:10:13 on dist-test-slave-95hh"
server_key: "fbefccf675f1baadd9fae41a1d89bab9"
server_key_iv: "2cc17133f01778092e37e802477df9bf"
server_key_version: "encryptionkey@0"
I20260923 23:10:13.614683 19784 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:13.626569 19784 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:13.628281 19784 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:13.628408 19784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:13.632038 19784 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:44529
I20260923 23:10:13.632072 19851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:44529 every 8 connection(s)
I20260923 23:10:13.632524 19784 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:13.633118 19852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:13.633893 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19784
I20260923 23:10:13.633968 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:13.634122 16874 external_mini_cluster.cc:1428] Setting key d1c5e6dc5fdb9087f3d0ce3037a39093
I20260923 23:10:13.636008 19852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: Bootstrap starting.
I20260923 23:10:13.636703 19852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:13.636977 19852 log.cc:824] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:13.637622 19852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: No bootstrap required, opened a new log
I20260923 23:10:13.639663 19852 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } }
I20260923 23:10:13.639801 19852 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:13.639822 19852 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb73aebb51dc4c1daa6bf155249dd7a2, State: Initialized, Role: FOLLOWER
I20260923 23:10:13.639933 19852 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } }
I20260923 23:10:13.640000 19852 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:13.640028 19852 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:13.640075 19852 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19771](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205013, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:13.640612 19852 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } }
I20260923 23:10:13.640705 19852 leader_election.cc:302] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb73aebb51dc4c1daa6bf155249dd7a2; no voters:
I20260923 23:10:13.640856 19852 leader_election.cc:288] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:13.640921 19857 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:13.641108 19857 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [term 1 LEADER]: Becoming Leader. State: Replica: bb73aebb51dc4c1daa6bf155249dd7a2, State: Running, Role: LEADER
I20260923 23:10:13.641108 19852 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:13.641250 19857 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } }
I20260923 23:10:13.641592 19858 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } } }
I20260923 23:10:13.641676 19858 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:13.641817 19859 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb73aebb51dc4c1daa6bf155249dd7a2. Latest consensus state: current_term: 1 leader_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb73aebb51dc4c1daa6bf155249dd7a2" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44529 } } }
I20260923 23:10:13.641875 19859 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2 [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:13.642817 19873 catalog_manager.cc:1558] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:13.642870 19873 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:13.642953 19865 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:13.643178 19865 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:13.644409 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:13.634570 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46890 (local address 127.16.122.190:44529)
0923 23:10:13.634747 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:13.634766 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:13.634797 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:10:13.635299 (+ 502us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:13.635308 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:13.635373 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:13.635522 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:13.636051 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:13.637316 (+ 1265us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:13.638043 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:13.638658 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:13.638774 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:13.640904 (+ 2130us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:13.640908 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:13.640909 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:13.640921 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:13.642949 (+ 2028us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:13.643306 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:13.643309 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:13.643310 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:13.643391 (+ 81us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:13.643567 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:13.643568 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:13.643568 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:13.643828 (+ 260us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:13.643974 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:13.644102 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:10:13.644239 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260923 23:10:13.644809 19865 catalog_manager.cc:1347] Generated new cluster ID: a59eadeb38274bceae364175d9e5b4ce
I20260923 23:10:13.644852 19865 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:13.645292 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:10:13.659080 19865 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:13.659701 19865 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:13.660372 19855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:13.656452 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46892 (local address 127.16.122.190:44529)
0923 23:10:13.656493 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:13.656495 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:13.656586 (+ 91us) server_negotiation.cc:407] Connection header received
0923 23:10:13.656634 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:13.656635 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:13.656645 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:13.656714 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:13.657085 (+ 371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:13.657868 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:13.658777 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:13.659122 (+ 345us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:13.659184 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:13.659903 (+ 719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:13.659904 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:13.659905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:13.660111 (+ 206us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:13.660131 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:13.660137 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:13.660196 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:13.660244 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 23:10:13.660293 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:13.666501 19865 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb73aebb51dc4c1daa6bf155249dd7a2: Generated new TSK 0
I20260923 23:10:13.666733 19865 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:13.675345 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19784
2026-09-23T23:10:13Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (1209 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19885](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19885](info): set up 2 sockets
Sep 23 23:10:13 dist-test-slave-95hh krb5kdc[19885](info): commencing operation
krb5kdc: starting...
W20260923 23:10:14.619982 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.004s sys 0.001s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:14 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:14.727739 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43999
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:43999
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:14.806803 19898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:14.807065 19898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:14.807099 19898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:14.808243 19898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:14.808288 19898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:14.808301 19898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:14.808312 19898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:14.808322 19898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:14.809721 19898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:43999
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43999
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:14.809916 19898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:14.810089 19898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:14.814635 19906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:14.814638 19904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:14.814687 19898 server_base.cc:1034] running on GCE node
W20260923 23:10:14.814644 19903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:14.815169 19898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:14.815466 19898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:14.816603 19898 hybrid_clock.cc:648] HybridClock initialized: now 1790205014816587 us; error 37 us; skew 500 ppm
Sep 23 23:10:14 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:14.818917 19898 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:14.819401 19898 webserver.cc:460] Webserver started at http://127.16.122.190:45105/ using document root <none> and password file <none>
I20260923 23:10:14.819594 19898 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:14.819639 19898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:14.819757 19898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:14.820809 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "5dd8290ad140ff40ee5d187820fbdb06"
server_key_iv: "c411d67ce9ea3b9e2d4004e0e3163277"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.821137 19898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "5dd8290ad140ff40ee5d187820fbdb06"
server_key_iv: "c411d67ce9ea3b9e2d4004e0e3163277"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.822335 19898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:14.822919 19913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:14.823098 19898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:14.823191 19898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "5dd8290ad140ff40ee5d187820fbdb06"
server_key_iv: "c411d67ce9ea3b9e2d4004e0e3163277"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.823254 19898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:14.836400 19898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:14.837991 19898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:14.838119 19898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:14.841822 19898 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:43999
I20260923 23:10:14.841874 19965 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:43999 every 8 connection(s)
I20260923 23:10:14.842167 19898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:14.843050 19966 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:14.845199 19966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071: Bootstrap starting.
I20260923 23:10:14.845726 19966 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:14.845970 19966 log.cc:824] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:14.846613 19966 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071: No bootstrap required, opened a new log
I20260923 23:10:14.848178 19966 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } }
I20260923 23:10:14.848325 19966 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:14.848345 19966 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3fe5dee3ebde4cd1ab84ff1ccee8b071, State: Initialized, Role: FOLLOWER
I20260923 23:10:14.848474 19966 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } }
I20260923 23:10:14.848538 19966 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:14.848567 19966 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:14.848613 19966 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:14.849354 19966 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } }
I20260923 23:10:14.849442 19966 leader_election.cc:302] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3fe5dee3ebde4cd1ab84ff1ccee8b071; no voters:
I20260923 23:10:14.849592 19966 leader_election.cc:288] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:14.849638 19969 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:14.849819 19969 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [term 1 LEADER]: Becoming Leader. State: Replica: 3fe5dee3ebde4cd1ab84ff1ccee8b071, State: Running, Role: LEADER
I20260923 23:10:14.849820 19966 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:14.849926 19969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } }
I20260923 23:10:14.850263 19971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3fe5dee3ebde4cd1ab84ff1ccee8b071. Latest consensus state: current_term: 1 leader_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } } }
I20260923 23:10:14.850284 19970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3fe5dee3ebde4cd1ab84ff1ccee8b071" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43999 } } }
I20260923 23:10:14.850425 19971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:14.850508 19970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:14.850855 19978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:14.851325 19978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:14.852020 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19898
I20260923 23:10:14.852118 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:14.852293 16874 external_mini_cluster.cc:1428] Setting key 77f20320fb6ad56ac47732520ad1f12c
I20260923 23:10:14.853020 19978 catalog_manager.cc:1347] Generated new cluster ID: c85051d5e6e349ac936f0d5c30b70301
I20260923 23:10:14.853072 19978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:14 dist-test-slave-95hh krb5kdc[19885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205014, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:14.863790 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:14.852867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59036 (local address 127.16.122.190:43999)
0923 23:10:14.853093 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:14.853101 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:14.853131 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:10:14.853239 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:14.853246 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:14.853310 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:14.853441 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:14.854166 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:14.855585 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:14.856619 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:14.857509 (+ 890us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:14.857608 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:14.860278 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:14.860282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:14.860282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:14.860295 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:14.862201 (+ 1906us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:14.862567 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:14.862569 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:14.862570 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:14.862672 (+ 102us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:14.862832 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:14.862832 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:14.862833 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:14.863158 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:14.863291 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:14.863400 (+ 109us) server_negotiation.cc:299] Negotiation successful
0923 23:10:14.863547 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188,"thread_start_us":78,"threads_started":1}
I20260923 23:10:14.873435 19978 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:14.873966 19978 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:14.877583 19978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3fe5dee3ebde4cd1ab84ff1ccee8b071: Generated new TSK 0
I20260923 23:10:14.877738 19978 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:14.902734 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:14.982648 19992 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:14.982885 19992 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:14.982921 19992 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:14.984123 19992 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:14.984167 19992 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:14.984203 19992 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:14.985664 19992 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.19992
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:14.985909 19992 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:14.986130 19992 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:14.991066 20000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:14.991096 19998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:14.991070 19997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:14.991314 19992 server_base.cc:1034] running on GCE node
I20260923 23:10:14.991534 19992 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:14.991765 19992 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:14.992910 19992 hybrid_clock.cc:648] HybridClock initialized: now 1790205014992893 us; error 34 us; skew 500 ppm
Sep 23 23:10:14 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:14.995333 19992 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:14.995770 19992 webserver.cc:460] Webserver started at http://127.16.122.129:42985/ using document root <none> and password file <none>
I20260923 23:10:14.995976 19992 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:14.996026 19992 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:14.996145 19992 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:14.997161 19992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "54240a4293434efebd3703760c098b4f"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "c5ede7ca53543328bcc70a2060dfafe7"
server_key_iv: "1977f4ff9a0184f4c8100f1ea24d6882"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.997499 19992 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "54240a4293434efebd3703760c098b4f"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "c5ede7ca53543328bcc70a2060dfafe7"
server_key_iv: "1977f4ff9a0184f4c8100f1ea24d6882"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.998729 19992 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:14.999446 20007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:14.999588 19992 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:14.999668 19992 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "54240a4293434efebd3703760c098b4f"
format_stamp: "Formatted at 2026-09-23 23:10:14 on dist-test-slave-95hh"
server_key: "c5ede7ca53543328bcc70a2060dfafe7"
server_key_iv: "1977f4ff9a0184f4c8100f1ea24d6882"
server_key_version: "encryptionkey@0"
I20260923 23:10:14.999727 19992 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:15.020277 19992 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:15.022073 19992 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:15.022217 19992 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:15.022433 19992 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:15.022748 19992 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:15.022782 19992 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.022810 19992 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:15.022827 19992 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.028416 19992 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:34713
I20260923 23:10:15.028468 20120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:34713 every 8 connection(s)
I20260923 23:10:15.028779 19992 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205014, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:15.037432 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.029485 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:43341 (local address 127.16.122.190:43999)
0923 23:10:15.029534 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:15.029537 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:15.029752 (+ 215us) server_negotiation.cc:407] Connection header received
0923 23:10:15.030228 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:15.030229 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:15.030241 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:15.030282 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:15.031236 (+ 954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.031733 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.032739 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.032918 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.032959 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.035147 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:15.035148 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:15.035148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:15.035155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:15.036371 (+ 1216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.036729 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.036729 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.036730 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.036774 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.036949 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.036950 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.036950 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.037043 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:15.037085 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:15.037297 (+ 212us) server_negotiation.cc:299] Negotiation successful
0923 23:10:15.037363 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:15.037591 20123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.029470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43999 (local address 127.16.122.129:43341)
0923 23:10:15.029631 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:15.029667 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:15.030149 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:15.030349 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:15.030352 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:15.030922 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:15.031163 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.031169 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.031813 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.031814 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.032671 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.032673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.033002 (+ 329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.033003 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.033116 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.033650 (+ 534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:15.033659 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:15.034997 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:15.036438 (+ 1441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.036444 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.036446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.036667 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.036817 (+ 150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.036817 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.036817 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.036908 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.037083 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:15.037086 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:15.037201 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 23:10:15.037280 (+ 79us) client_negotiation.cc:240] Negotiation successful
0923 23:10:15.037402 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":77,"threads_started":1}
I20260923 23:10:15.037978 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 19992
I20260923 23:10:15.038080 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:15.038091 20121 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43999
I20260923 23:10:15.038173 20121 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:15.038223 16874 external_mini_cluster.cc:1428] Setting key efc7cde0797e190296ed200a4af585cd
I20260923 23:10:15.038314 20121 heartbeater.cc:511] Master 127.16.122.190:43999 requested a full tablet report, sending...
I20260923 23:10:15.038797 19929 ts_manager.cc:194] Registered new tserver with Master: 54240a4293434efebd3703760c098b4f (127.16.122.129:34713)
I20260923 23:10:15.039636 19929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:43341
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:15.062047 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:15.140698 20126 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:15.140906 20126 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:15.140930 20126 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:15.142056 20126 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:15.142097 20126 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:15.142122 20126 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:15.143605 20126 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20126
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:15.143824 20126 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:15.144037 20126 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:15.148782 20134 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:15.148788 20132 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:15.148943 20126 server_base.cc:1034] running on GCE node
W20260923 23:10:15.149022 20131 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:15.149205 20126 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:15.149434 20126 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:15.150583 20126 hybrid_clock.cc:648] HybridClock initialized: now 1790205015150572 us; error 41 us; skew 500 ppm
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:15.152998 20126 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:15.153424 20126 webserver.cc:460] Webserver started at http://127.16.122.130:36591/ using document root <none> and password file <none>
I20260923 23:10:15.153627 20126 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:15.153669 20126 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:15.153789 20126 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:15.154794 20126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "5385f3cb96cc41289c7306bb4f8fa1a3"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "00b13cbde05dba5656ef20f5159436f3"
server_key_iv: "c69286a7698eb0df301b03b2ae96c6e1"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.155134 20126 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "5385f3cb96cc41289c7306bb4f8fa1a3"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "00b13cbde05dba5656ef20f5159436f3"
server_key_iv: "c69286a7698eb0df301b03b2ae96c6e1"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.156308 20126 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:15.156950 20141 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.157091 20126 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:15.157161 20126 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "5385f3cb96cc41289c7306bb4f8fa1a3"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "00b13cbde05dba5656ef20f5159436f3"
server_key_iv: "c69286a7698eb0df301b03b2ae96c6e1"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.157226 20126 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:15.169378 20126 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:15.171329 20126 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:15.171459 20126 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:15.171657 20126 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:15.171947 20126 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:15.171980 20126 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.172009 20126 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:15.172026 20126 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.177332 20126 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:36181
I20260923 23:10:15.177383 20254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:36181 every 8 connection(s)
I20260923 23:10:15.177660 20126 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:15.185931 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.178266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:49901 (local address 127.16.122.190:43999)
0923 23:10:15.178303 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:15.178305 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:15.178528 (+ 223us) server_negotiation.cc:407] Connection header received
0923 23:10:15.179013 (+ 485us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:15.179015 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:15.179027 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:15.179093 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:15.179970 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.180478 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.181443 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.181639 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.181693 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.183633 (+ 1940us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:15.183635 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:15.183635 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:15.183643 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:15.184725 (+ 1082us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.185085 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.185087 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.185087 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.185138 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.185343 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.185344 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.185344 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.185457 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:15.185511 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:15.185756 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 23:10:15.185839 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:10:15.186108 20257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.178276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43999 (local address 127.16.122.130:49901)
0923 23:10:15.178418 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:15.178453 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:15.178917 (+ 464us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:15.179163 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:15.179167 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:15.179694 (+ 527us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:15.179877 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.179881 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.180557 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.180558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.181365 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.181367 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.181704 (+ 337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.181705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.181825 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.182266 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:15.182274 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:15.183549 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:15.184798 (+ 1249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.184804 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.184806 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.185021 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.185193 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.185194 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.185194 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.185291 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.185510 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:15.185516 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:15.185645 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 23:10:15.185731 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 23:10:15.185882 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":74,"threads_started":1}
I20260923 23:10:15.186537 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20126
I20260923 23:10:15.186604 20255 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43999
I20260923 23:10:15.186635 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:15.186699 20255 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:15.186846 16874 external_mini_cluster.cc:1428] Setting key 2a9b1697ca77907c7cc50adf3fbe1cd9
I20260923 23:10:15.186898 20255 heartbeater.cc:511] Master 127.16.122.190:43999 requested a full tablet report, sending...
I20260923 23:10:15.187294 19930 ts_manager.cc:194] Registered new tserver with Master: 5385f3cb96cc41289c7306bb4f8fa1a3 (127.16.122.130:36181)
I20260923 23:10:15.187733 19930 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:49901
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:15.209712 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38497
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:15.288508 20260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:15.288720 20260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:15.288745 20260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:15.289863 20260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:15.289901 20260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:15.289925 20260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:15.291334 20260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38497
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43999
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20260
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:15.291527 20260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:15.291702 20260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:15.296372 20266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:15.296360 20265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:15.296350 20268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:15.296563 20260 server_base.cc:1034] running on GCE node
I20260923 23:10:15.296767 20260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:15.296995 20260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:15.298127 20260 hybrid_clock.cc:648] HybridClock initialized: now 1790205015298112 us; error 30 us; skew 500 ppm
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:15.300423 20260 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:15.300815 20260 webserver.cc:460] Webserver started at http://127.16.122.131:43873/ using document root <none> and password file <none>
I20260923 23:10:15.301010 20260 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:15.301054 20260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:15.301177 20260 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:15.302186 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "2678a99e6fd84be3bc72e55c0eb0e46b"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "ddc43662f486b1cbe6516c4a0bb03728"
server_key_iv: "d5ee15e9e904e27266630ab85708a14d"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.302541 20260 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "2678a99e6fd84be3bc72e55c0eb0e46b"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "ddc43662f486b1cbe6516c4a0bb03728"
server_key_iv: "d5ee15e9e904e27266630ab85708a14d"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.303810 20260 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.002s
I20260923 23:10:15.304440 20275 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.304576 20260 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:15.304653 20260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "2678a99e6fd84be3bc72e55c0eb0e46b"
format_stamp: "Formatted at 2026-09-23 23:10:15 on dist-test-slave-95hh"
server_key: "ddc43662f486b1cbe6516c4a0bb03728"
server_key_iv: "d5ee15e9e904e27266630ab85708a14d"
server_key_version: "encryptionkey@0"
I20260923 23:10:15.304714 20260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:15.316613 20260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:15.318498 20260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:15.318629 20260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:15.318842 20260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:15.319209 20260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:15.319245 20260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.319274 20260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:15.319299 20260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:15.324772 20260 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:40711
I20260923 23:10:15.324815 20388 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:40711 every 8 connection(s)
I20260923 23:10:15.325142 20260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:15.333855 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.325803 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:48165 (local address 127.16.122.190:43999)
0923 23:10:15.325847 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:15.325850 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:15.326045 (+ 195us) server_negotiation.cc:407] Connection header received
0923 23:10:15.326617 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:15.326619 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:15.326631 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:15.326699 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:15.327730 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.328220 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.329198 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.329393 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.329448 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.331535 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:15.331537 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:15.331537 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:15.331544 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:15.332733 (+ 1189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.333111 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.333111 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.333112 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.333158 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.333386 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.333387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.333387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.333482 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:15.333525 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:15.333732 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 23:10:15.333789 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:15.334041 20391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.325781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43999 (local address 127.16.122.131:48165)
0923 23:10:15.325920 (+ 139us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:15.325959 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:15.326509 (+ 550us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:15.326781 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:15.326785 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:15.327432 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:15.327630 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.327636 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.328301 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.328302 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.329117 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:15.329119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.329462 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.329463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:15.329580 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.330060 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:15.330068 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:15.331451 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:15.332819 (+ 1368us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.332825 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.332827 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.333044 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.333212 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:15.333213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:15.333213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:15.333341 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:15.333523 (+ 182us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:15.333526 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:15.333639 (+ 113us) client_negotiation.cc:769] Sending connection context
0923 23:10:15.333716 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:10:15.333832 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":71,"threads_started":1}
I20260923 23:10:15.334173 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20260
I20260923 23:10:15.334280 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:15.334491 16874 external_mini_cluster.cc:1428] Setting key f7ee1c48deac9be1cc7b4660219a1d02
I20260923 23:10:15.334555 20389 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43999
I20260923 23:10:15.334646 20389 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:15.334820 20389 heartbeater.cc:511] Master 127.16.122.190:43999 requested a full tablet report, sending...
I20260923 23:10:15.335203 19929 ts_manager.cc:194] Registered new tserver with Master: 2678a99e6fd84be3bc72e55c0eb0e46b (127.16.122.131:40711)
I20260923 23:10:15.335578 19929 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:48165
I20260923 23:10:15.346187 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[19885](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205015, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:15.366914 19989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:15.360588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59046 (local address 127.16.122.190:43999)
0923 23:10:15.360643 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:15.360645 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:15.360757 (+ 112us) server_negotiation.cc:407] Connection header received
0923 23:10:15.360817 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:15.360818 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:15.360829 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:15.360877 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:15.361399 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.361878 (+ 479us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.362537 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:15.362702 (+ 165us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:15.362743 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:15.364760 (+ 2017us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:15.364762 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:15.364762 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:15.364768 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:15.366102 (+ 1334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.366361 (+ 259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.366362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.366362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.366406 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:15.366523 (+ 117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:15.366523 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:15.366523 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:15.366651 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:15.366725 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:15.366793 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 23:10:15.366849 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
W20260923 23:10:15.367157 19929 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59046
I20260923 23:10:15.367540 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19992
I20260923 23:10:15.371435 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20126
I20260923 23:10:15.375290 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20260
I20260923 23:10:15.378901 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 19898
2026-09-23T23:10:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (1705 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[20401](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[20401](info): set up 2 sockets
Sep 23 23:10:15 dist-test-slave-95hh krb5kdc[20401](info): commencing operation
krb5kdc: starting...
W20260923 23:10:16.331543 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.934s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:16.447652 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34795
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34795
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:16.525352 20414 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:16.525561 20414 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:16.525583 20414 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:16.526731 20414 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:16.526772 20414 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:16.526789 20414 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:16.526800 20414 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:16.526811 20414 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:16.528303 20414 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34795
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20414
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:16.528496 20414 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:16.528674 20414 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:16.533491 20420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.533560 20414 server_base.cc:1034] running on GCE node
W20260923 23:10:16.533500 20422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:16.533500 20419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.533931 20414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:16.534230 20414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:16.535368 20414 hybrid_clock.cc:648] HybridClock initialized: now 1790205016535352 us; error 37 us; skew 500 ppm
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:16.540964 20414 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:16.541407 20414 webserver.cc:460] Webserver started at http://127.16.122.190:33279/ using document root <none> and password file <none>
I20260923 23:10:16.541663 20414 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:16.541714 20414 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:16.541839 20414 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:16.542868 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "94fb13aa218747ad9813410317c5ae36"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "c915fcf9abfe9808f0f7b4f3b34ed2eb"
server_key_iv: "f09c8e71c8c6b94d8cb84c168eaede74"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.543224 20414 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "94fb13aa218747ad9813410317c5ae36"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "c915fcf9abfe9808f0f7b4f3b34ed2eb"
server_key_iv: "f09c8e71c8c6b94d8cb84c168eaede74"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.544487 20414 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:16.545163 20429 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.545308 20414 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:16.545389 20414 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "94fb13aa218747ad9813410317c5ae36"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "c915fcf9abfe9808f0f7b4f3b34ed2eb"
server_key_iv: "f09c8e71c8c6b94d8cb84c168eaede74"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.545449 20414 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:16.583880 20414 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:16.585923 20414 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:16.586063 20414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:16.589835 20414 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34795
I20260923 23:10:16.589888 20481 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34795 every 8 connection(s)
I20260923 23:10:16.590358 20414 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:16.591017 20482 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:16.592334 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20414
I20260923 23:10:16.592417 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:16.592571 16874 external_mini_cluster.cc:1428] Setting key e33fd6d381d4b222dadd9ed99964f8c1
I20260923 23:10:16.593629 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36: Bootstrap starting.
I20260923 23:10:16.594190 20482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:16.594453 20482 log.cc:824] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:16.595103 20482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36: No bootstrap required, opened a new log
I20260923 23:10:16.596830 20482 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } }
I20260923 23:10:16.596982 20482 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:16.597015 20482 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94fb13aa218747ad9813410317c5ae36, State: Initialized, Role: FOLLOWER
I20260923 23:10:16.597141 20482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } }
I20260923 23:10:16.597222 20482 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:16.597287 20482 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:16.597332 20482 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:16.597874 20482 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } }
I20260923 23:10:16.597965 20482 leader_election.cc:302] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94fb13aa218747ad9813410317c5ae36; no voters:
I20260923 23:10:16.598114 20482 leader_election.cc:288] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:16.598153 20487 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:16.598294 20487 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [term 1 LEADER]: Becoming Leader. State: Replica: 94fb13aa218747ad9813410317c5ae36, State: Running, Role: LEADER
I20260923 23:10:16.598379 20482 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:16.598448 20487 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } }
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:16.598824 20487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94fb13aa218747ad9813410317c5ae36. Latest consensus state: current_term: 1 leader_uuid: "94fb13aa218747ad9813410317c5ae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } } }
I20260923 23:10:16.598903 20487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:16.599073 20489 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94fb13aa218747ad9813410317c5ae36" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94fb13aa218747ad9813410317c5ae36" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34795 } } }
I20260923 23:10:16.599133 20489 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:16.599227 20495 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:16.599989 20495 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:16.601652 20495 catalog_manager.cc:1347] Generated new cluster ID: 59cf9d294261483aa05d28747138b57a
I20260923 23:10:16.601739 20495 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:16.602810 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:16.593055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42592 (local address 127.16.122.190:34795)
0923 23:10:16.593229 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:16.593238 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:16.593262 (+ 24us) server_negotiation.cc:407] Connection header received
0923 23:10:16.593660 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:16.593666 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:16.593714 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:16.593855 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:16.594484 (+ 629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.595500 (+ 1016us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.596213 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.596632 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.596727 (+ 95us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:16.599282 (+ 2555us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:16.599286 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:16.599286 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:16.599298 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:16.601225 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.601655 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.601658 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.601659 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.601776 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.601987 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.601988 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.601988 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.602219 (+ 231us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:16.602362 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:16.602478 (+ 116us) server_negotiation.cc:299] Negotiation successful
0923 23:10:16.602607 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":74,"threads_started":1}
I20260923 23:10:16.626940 20495 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:16.627573 20495 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:16.630579 20495 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94fb13aa218747ad9813410317c5ae36: Generated new TSK 0
I20260923 23:10:16.630757 20495 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:16.659687 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:16.771896 20508 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:16.772105 20508 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:16.772130 20508 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:16.773289 20508 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:16.773332 20508 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:16.773358 20508 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:16.774832 20508 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20508
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:16.775090 20508 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:16.775326 20508 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:16.780224 20513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.780337 20508 server_base.cc:1034] running on GCE node
W20260923 23:10:16.780220 20514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:16.780418 20516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.780581 20508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:16.780794 20508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:16.781935 20508 hybrid_clock.cc:648] HybridClock initialized: now 1790205016781923 us; error 27 us; skew 500 ppm
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:16.784883 20508 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:16.785619 20508 webserver.cc:460] Webserver started at http://127.16.122.129:46635/ using document root <none> and password file <none>
I20260923 23:10:16.785867 20508 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:16.785925 20508 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:16.786072 20508 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:16.787703 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "f8dfb2b34f0449e9b891cb2168746f94"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "0ef7dde1ab41b2f533e5c0a5423b2769"
server_key_iv: "0333c7d63acaee2cce93ccb00bee04c5"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.788259 20508 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "f8dfb2b34f0449e9b891cb2168746f94"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "0ef7dde1ab41b2f533e5c0a5423b2769"
server_key_iv: "0333c7d63acaee2cce93ccb00bee04c5"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.790053 20508 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 23:10:16.791062 20523 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.791791 20508 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:16.791884 20508 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "f8dfb2b34f0449e9b891cb2168746f94"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "0ef7dde1ab41b2f533e5c0a5423b2769"
server_key_iv: "0333c7d63acaee2cce93ccb00bee04c5"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.791957 20508 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:16.801625 20508 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:16.804363 20508 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:16.804512 20508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:16.804749 20508 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:16.805109 20508 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:16.805157 20508 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.805187 20508 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:16.805212 20508 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.811968 20508 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42111
I20260923 23:10:16.812009 20636 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42111 every 8 connection(s)
I20260923 23:10:16.812397 20508 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:16.815126 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20508
I20260923 23:10:16.815236 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:16.815430 16874 external_mini_cluster.cc:1428] Setting key 24ddf7cb816b98df19cfea8f68110d43
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:16.823462 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:16.813117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:40821 (local address 127.16.122.190:34795)
0923 23:10:16.813183 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:16.813185 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:16.813348 (+ 163us) server_negotiation.cc:407] Connection header received
0923 23:10:16.814478 (+ 1130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:16.814480 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:16.814492 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:16.814576 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:16.815394 (+ 818us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.816132 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.817187 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.817407 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.817462 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:16.820245 (+ 2783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:16.820247 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:16.820247 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:16.820253 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:16.821862 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.822338 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.822340 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.822340 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.822404 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.822679 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.822679 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.822680 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.822818 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:16.822888 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:16.823267 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 23:10:16.823361 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:10:16.823647 20639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:16.813065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34795 (local address 127.16.122.129:40821)
0923 23:10:16.813221 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:16.813262 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:16.814367 (+ 1105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:16.814648 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:16.814652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:16.814993 (+ 341us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:16.815312 (+ 319us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:16.815318 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.816228 (+ 910us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.816228 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:16.817111 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:16.817114 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.817473 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.817474 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:16.817617 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:16.818763 (+ 1146us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:16.818771 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:16.820154 (+ 1383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:16.821953 (+ 1799us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:16.821961 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:16.821964 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:16.822257 (+ 293us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:16.822471 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:16.822472 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:16.822472 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:16.822612 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:16.822888 (+ 276us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:16.822895 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:16.823090 (+ 195us) client_negotiation.cc:769] Sending connection context
0923 23:10:16.823240 (+ 150us) client_negotiation.cc:240] Negotiation successful
0923 23:10:16.823412 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":87,"threads_started":1}
I20260923 23:10:16.824268 20637 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34795
I20260923 23:10:16.824380 20637 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:16.824584 20637 heartbeater.cc:511] Master 127.16.122.190:34795 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:10:16.825141 20446 ts_manager.cc:194] Registered new tserver with Master: f8dfb2b34f0449e9b891cb2168746f94 (127.16.122.129:42111)
I20260923 23:10:16.825989 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:40821
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:16.839747 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:16.919701 20642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:16.919912 20642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:16.919942 20642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:16.921090 20642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:16.921132 20642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:16.921167 20642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:16.922637 20642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20642
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:16.922873 20642 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:16.923107 20642 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:16.927812 20650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:16.927788 20648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.927918 20642 server_base.cc:1034] running on GCE node
W20260923 23:10:16.927800 20647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:16.928197 20642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:16.928440 20642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:16.929615 20642 hybrid_clock.cc:648] HybridClock initialized: now 1790205016929603 us; error 26 us; skew 500 ppm
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:16.932066 20642 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:16.932529 20642 webserver.cc:460] Webserver started at http://127.16.122.130:37573/ using document root <none> and password file <none>
I20260923 23:10:16.932736 20642 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:16.932782 20642 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:16.932904 20642 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:16.933938 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "2314cf9eb8414193881cd8c3fdbe2ae9"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "f03c83f79ae3d4df5be9f08ba031c770"
server_key_iv: "73931f0facfc4fdd9b5c20e04759fe2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.934247 20642 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "2314cf9eb8414193881cd8c3fdbe2ae9"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "f03c83f79ae3d4df5be9f08ba031c770"
server_key_iv: "73931f0facfc4fdd9b5c20e04759fe2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.935552 20642 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:16.936236 20657 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.936384 20642 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:16.936465 20642 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "2314cf9eb8414193881cd8c3fdbe2ae9"
format_stamp: "Formatted at 2026-09-23 23:10:16 on dist-test-slave-95hh"
server_key: "f03c83f79ae3d4df5be9f08ba031c770"
server_key_iv: "73931f0facfc4fdd9b5c20e04759fe2e"
server_key_version: "encryptionkey@0"
I20260923 23:10:16.936523 20642 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:16.948879 20642 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:16.950719 20642 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:16.950851 20642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:16.951143 20642 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:16.951447 20642 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:16.951479 20642 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.951509 20642 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:16.951529 20642 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:16.956843 20642 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:34363
I20260923 23:10:16.956899 20770 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:34363 every 8 connection(s)
I20260923 23:10:16.957186 20642 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:16 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205016, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:16.964246 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20642
I20260923 23:10:16.964355 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:16.964542 16874 external_mini_cluster.cc:1428] Setting key da16a9ddb0c9fef571c3daa18a1bed5a
I20260923 23:10:16.966526 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:16.957829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:52187 (local address 127.16.122.190:34795)
0923 23:10:16.957880 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:16.957882 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:16.958083 (+ 201us) server_negotiation.cc:407] Connection header received
0923 23:10:16.958949 (+ 866us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:16.958950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:16.958958 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:16.959021 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:16.959663 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.960144 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.961221 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.961407 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.961451 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:16.963639 (+ 2188us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:16.963641 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:16.963641 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:16.963649 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:16.965247 (+ 1598us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.965621 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.965622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.965622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.965681 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:16.965914 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:16.965916 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:16.965916 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:16.966030 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:16.966085 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:16.966351 (+ 266us) server_negotiation.cc:299] Negotiation successful
0923 23:10:16.966431 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:10:16.966738 20773 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:16.957800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34795 (local address 127.16.122.130:52187)
0923 23:10:16.957964 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:16.958005 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:16.958869 (+ 864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:16.959117 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:16.959122 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:16.959400 (+ 278us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:16.959593 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:16.959598 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.960244 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.960245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:16.961133 (+ 888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:16.961135 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:16.961498 (+ 363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:16.961499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:16.961627 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:16.962103 (+ 476us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:16.962112 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:16.963542 (+ 1430us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:16.965306 (+ 1764us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:16.965314 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:16.965316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:16.965552 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:16.965734 (+ 182us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:16.965735 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:16.965735 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:16.965847 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:16.966083 (+ 236us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:16.966089 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:16.966233 (+ 144us) client_negotiation.cc:769] Sending connection context
0923 23:10:16.966323 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:10:16.966498 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":79,"threads_started":1}
I20260923 23:10:16.967186 20771 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34795
I20260923 23:10:16.967291 20771 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:16.967451 20771 heartbeater.cc:511] Master 127.16.122.190:34795 requested a full tablet report, sending...
I20260923 23:10:16.967799 20446 ts_manager.cc:194] Registered new tserver with Master: 2314cf9eb8414193881cd8c3fdbe2ae9 (127.16.122.130:34363)
I20260923 23:10:16.968161 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:52187
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:16.987191 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:17.065443 20776 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:17.065671 20776 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:17.065703 20776 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:17.066874 20776 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:17.066913 20776 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:17.066948 20776 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:17.068501 20776 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34795
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20776
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:17.068734 20776 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:17.068938 20776 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:17.073661 20784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:17.073730 20776 server_base.cc:1034] running on GCE node
W20260923 23:10:17.073642 20781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:17.073645 20782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:17.074055 20776 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:17.074281 20776 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:17.075412 20776 hybrid_clock.cc:648] HybridClock initialized: now 1790205017075397 us; error 26 us; skew 500 ppm
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205017, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:17.077726 20776 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:17.078155 20776 webserver.cc:460] Webserver started at http://127.16.122.131:38919/ using document root <none> and password file <none>
I20260923 23:10:17.078346 20776 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:17.078388 20776 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:17.078492 20776 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:17.079532 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "3e98296675c64ce4b5936bda01883ff2"
format_stamp: "Formatted at 2026-09-23 23:10:17 on dist-test-slave-95hh"
server_key: "be5eb413cdbd89c2b62b3d25337c9b22"
server_key_iv: "28ca987a58d0d69129c65a1ee518597f"
server_key_version: "encryptionkey@0"
I20260923 23:10:17.079840 20776 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "3e98296675c64ce4b5936bda01883ff2"
format_stamp: "Formatted at 2026-09-23 23:10:17 on dist-test-slave-95hh"
server_key: "be5eb413cdbd89c2b62b3d25337c9b22"
server_key_iv: "28ca987a58d0d69129c65a1ee518597f"
server_key_version: "encryptionkey@0"
I20260923 23:10:17.081023 20776 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:17.081710 20791 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:17.081875 20776 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:17.081955 20776 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "3e98296675c64ce4b5936bda01883ff2"
format_stamp: "Formatted at 2026-09-23 23:10:17 on dist-test-slave-95hh"
server_key: "be5eb413cdbd89c2b62b3d25337c9b22"
server_key_iv: "28ca987a58d0d69129c65a1ee518597f"
server_key_version: "encryptionkey@0"
I20260923 23:10:17.082008 20776 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:17.092986 20776 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:17.095019 20776 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:17.095180 20776 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:17.095366 20776 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:17.095614 20776 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:17.095649 20776 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:17.095672 20776 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:17.095688 20776 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:17.101122 20776 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:34745
I20260923 23:10:17.101184 20904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:34745 every 8 connection(s)
I20260923 23:10:17.101490 20776 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205017, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:17.109876 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:17.102151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:60859 (local address 127.16.122.190:34795)
0923 23:10:17.102195 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:17.102197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:17.102366 (+ 169us) server_negotiation.cc:407] Connection header received
0923 23:10:17.103331 (+ 965us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:17.103332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:17.103342 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:17.103400 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:17.103962 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.104482 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.105387 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.105582 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.105640 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:17.107571 (+ 1931us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:17.107573 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:17.107573 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:17.107580 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:17.108863 (+ 1283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.109184 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.109184 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.109185 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.109231 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.109402 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.109402 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.109403 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.109497 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:17.109543 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:17.109751 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 23:10:17.109811 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:17.110041 20907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:17.102131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34795 (local address 127.16.122.131:60859)
0923 23:10:17.102265 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:17.102300 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:17.103214 (+ 914us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:17.103454 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:17.103457 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:17.103708 (+ 251us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:17.103891 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:17.103895 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.104545 (+ 650us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.104545 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:17.105316 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:17.105318 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.105641 (+ 323us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.105641 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:17.105735 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:17.106153 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:17.106160 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:17.107469 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:17.108930 (+ 1461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:17.108933 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:17.108935 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:17.109134 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:17.109274 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:17.109274 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:17.109274 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:17.109362 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:17.109539 (+ 177us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:17.109542 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:17.109657 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 23:10:17.109735 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:10:17.109850 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":74,"threads_started":1}
I20260923 23:10:17.110452 20905 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34795
I20260923 23:10:17.110550 20905 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:17.110700 20905 heartbeater.cc:511] Master 127.16.122.190:34795 requested a full tablet report, sending...
I20260923 23:10:17.111044 20446 ts_manager.cc:194] Registered new tserver with Master: 3e98296675c64ce4b5936bda01883ff2 (127.16.122.131:34745)
I20260923 23:10:17.111393 20446 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:60859
I20260923 23:10:17.111596 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20776
I20260923 23:10:17.111685 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:17.111864 16874 external_mini_cluster.cc:1428] Setting key 94749e39e797a3e89c01170f1956b108
I20260923 23:10:17.112568 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20401](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:17.133754 20912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:17.126602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44570 (local address 127.16.122.129:42111)
0923 23:10:17.126762 (+ 160us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:17.126767 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:17.126778 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:17.126813 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:17.126814 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:17.126828 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:17.126909 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:17.127466 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.128184 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.128877 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.129213 (+ 336us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.129272 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:17.131384 (+ 2112us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:17.131392 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:17.131392 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:17.131399 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:17.132813 (+ 1414us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.133143 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.133146 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.133146 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.133207 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.133334 (+ 127us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.133334 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.133334 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.133451 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:17.133545 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:17.133599 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 23:10:17.133655 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":81,"threads_started":1}
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20401](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205017, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:17.140766 20485 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:17.134917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42606 (local address 127.16.122.190:34795)
0923 23:10:17.134961 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:17.134962 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:17.134974 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:17.135075 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:17.135076 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:17.135086 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:17.135135 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:17.135520 (+ 385us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.135989 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.136572 (+ 583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:17.136745 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:17.136788 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:17.138603 (+ 1815us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:17.138605 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:17.138605 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:17.138611 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:17.139870 (+ 1259us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.140176 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.140176 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.140177 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.140230 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:17.140358 (+ 128us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:17.140359 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:17.140359 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:17.140496 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:17.140564 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:17.140643 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 23:10:17.140701 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 23:10:17.141003 20446 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42606
I20260923 23:10:17.141372 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20508
I20260923 23:10:17.145918 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20642
I20260923 23:10:17.150111 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20776
I20260923 23:10:17.154177 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20414
2026-09-23T23:10:17Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (1774 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20917](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20917](info): set up 2 sockets
Sep 23 23:10:17 dist-test-slave-95hh krb5kdc[20917](info): commencing operation
krb5kdc: starting...
W20260923 23:10:18.095552 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:18Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:18.204025 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43565
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:43565
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:18.281808 20930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:18.282027 20930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:18.282049 20930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:18.283203 20930 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:18.283243 20930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:18.283257 20930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:18.283267 20930 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:18.283282 20930 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:18.284914 20930 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:43565
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43565
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.20930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:18.285123 20930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:18.285302 20930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:18.289825 20936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.289839 20938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.289844 20935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:18.289985 20930 server_base.cc:1034] running on GCE node
I20260923 23:10:18.290241 20930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:18.290529 20930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:18.291667 20930 hybrid_clock.cc:648] HybridClock initialized: now 1790205018291651 us; error 29 us; skew 500 ppm
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:18.293947 20930 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:18.294255 20930 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:18.294303 20930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:18.294422 20930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:18.295481 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "a33d8e59f17b4c998d172a7e6a35b04f"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "5544d6dc3a0303a13225a8756a0166d7"
server_key_iv: "99de3bc1bf6cf7342337cba9364c0820"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.295811 20930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "a33d8e59f17b4c998d172a7e6a35b04f"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "5544d6dc3a0303a13225a8756a0166d7"
server_key_iv: "99de3bc1bf6cf7342337cba9364c0820"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.297005 20930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:18.297655 20944 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.297797 20930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:18.297878 20930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "a33d8e59f17b4c998d172a7e6a35b04f"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "5544d6dc3a0303a13225a8756a0166d7"
server_key_iv: "99de3bc1bf6cf7342337cba9364c0820"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.297937 20930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:18.313277 20930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:18.315021 20930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:18.315205 20930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:18.318764 20930 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:43565
I20260923 23:10:18.318792 20996 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:43565 every 8 connection(s)
I20260923 23:10:18.319097 20930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:18.320003 20997 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:18.322259 20997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: Bootstrap starting.
I20260923 23:10:18.322858 20997 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:18.323148 20997 log.cc:824] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:18.323804 20997 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: No bootstrap required, opened a new log
I20260923 23:10:18.325366 20997 raft_consensus.cc:348] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } }
I20260923 23:10:18.325510 20997 raft_consensus.cc:374] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:18.325542 20997 raft_consensus.cc:729] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33d8e59f17b4c998d172a7e6a35b04f, State: Initialized, Role: FOLLOWER
I20260923 23:10:18.325690 20997 consensus_queue.cc:260] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } }
I20260923 23:10:18.325748 20997 raft_consensus.cc:388] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:18.325771 20997 raft_consensus.cc:482] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:18.325824 20997 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:18.326617 20997 raft_consensus.cc:504] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } }
I20260923 23:10:18.326707 20997 leader_election.cc:302] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a33d8e59f17b4c998d172a7e6a35b04f; no voters:
I20260923 23:10:18.326900 20997 leader_election.cc:288] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:18.326951 21000 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:18.327126 21000 raft_consensus.cc:686] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [term 1 LEADER]: Becoming Leader. State: Replica: a33d8e59f17b4c998d172a7e6a35b04f, State: Running, Role: LEADER
I20260923 23:10:18.327179 20997 sys_catalog.cc:564] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:18.327226 21000 consensus_queue.cc:237] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } }
I20260923 23:10:18.327567 21001 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } } }
I20260923 23:10:18.327572 21002 sys_catalog.cc:455] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [sys.catalog]: SysCatalogTable state changed. Reason: New leader a33d8e59f17b4c998d172a7e6a35b04f. Latest consensus state: current_term: 1 leader_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a33d8e59f17b4c998d172a7e6a35b04f" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43565 } } }
I20260923 23:10:18.327657 21001 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:18.327685 21002 sys_catalog.cc:458] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:18.327860 21009 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:18.327878 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 20930
I20260923 23:10:18.327961 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:18.328161 16874 external_mini_cluster.cc:1428] Setting key 7f6efcf61029298b180f825f402b4cfd
I20260923 23:10:18.328380 21009 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:18.330482 21009 catalog_manager.cc:1347] Generated new cluster ID: e23c0ceef9af46738ac2a19155e800dc
I20260923 23:10:18.330535 21009 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:18.337755 21009 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:18.337815 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.328794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.16.122.190:43565)
0923 23:10:18.328961 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.328971 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.329210 (+ 239us) server_negotiation.cc:407] Connection header received
0923 23:10:18.329465 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.329477 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.329547 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.330424 (+ 877us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:18.330528 (+ 104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.331559 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.332260 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.332693 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.332783 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.334881 (+ 2098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:18.334884 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:18.334885 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:18.334897 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:18.336413 (+ 1516us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.336717 (+ 304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.336719 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.336720 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.336807 (+ 87us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.336990 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.336991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.336991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.337217 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:18.337363 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.337455 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.337607 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":79,"threads_started":1}
I20260923 23:10:18.338315 21009 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:18.345479 21009 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: Generated new TSK 0
I20260923 23:10:18.345690 21009 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:18.374511 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:18.454564 21023 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:18.454763 21023 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:18.454784 21023 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:18.455926 21023 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:18.455967 21023 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:18.455992 21023 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:18.457458 21023 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21023
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:18.457675 21023 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:18.457890 21023 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:18.462548 21031 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.462548 21028 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:18.462634 21023 server_base.cc:1034] running on GCE node
W20260923 23:10:18.462548 21029 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:18.462951 21023 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:18.463220 21023 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:18.464398 21023 hybrid_clock.cc:648] HybridClock initialized: now 1790205018464380 us; error 28 us; skew 500 ppm
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:18.466931 21023 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:18.467314 21023 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:18.467368 21023 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:18.467468 21023 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:18.468509 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "fe45358b4f664370b64fdb0ae635b873"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "7af638aa99aa064f44699617fa91ea36"
server_key_iv: "693c2574137ed06961a950d1e470ade7"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.468840 21023 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "fe45358b4f664370b64fdb0ae635b873"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "7af638aa99aa064f44699617fa91ea36"
server_key_iv: "693c2574137ed06961a950d1e470ade7"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.470095 21023 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:18.470760 21037 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.470908 21023 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:18.470984 21023 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "fe45358b4f664370b64fdb0ae635b873"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "7af638aa99aa064f44699617fa91ea36"
server_key_iv: "693c2574137ed06961a950d1e470ade7"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.471071 21023 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:18.489960 21023 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:18.491760 21023 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:18.491866 21023 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:18.492054 21023 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:18.492319 21023 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:18.492348 21023 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.492368 21023 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:18.492384 21023 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.497632 21023 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42565
I20260923 23:10:18.497691 21150 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42565 every 8 connection(s)
I20260923 23:10:18.497939 21023 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:18.499316 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21023
I20260923 23:10:18.499397 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:18.499559 16874 external_mini_cluster.cc:1428] Setting key 50dc1280b3802c656e43bc3dd0bbc01c
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:10:18.507344 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.498636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:36459 (local address 127.16.122.190:43565)
0923 23:10:18.498695 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.498697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.498891 (+ 194us) server_negotiation.cc:407] Connection header received
0923 23:10:18.499498 (+ 607us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.499499 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.499511 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.499560 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:18.500683 (+ 1123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.501190 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.502239 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.502530 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.502594 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.504786 (+ 2192us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:18.504787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:18.504788 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:18.504794 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:18.505979 (+ 1185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.506423 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.506424 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.506424 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.506466 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.506715 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.506716 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.506716 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.506808 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:18.506852 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.507214 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.507278 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:18.507596 21153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.498604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43565 (local address 127.16.122.129:36459)
0923 23:10:18.498767 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.498803 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.499371 (+ 568us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.499632 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.499636 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.500000 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:18.500595 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.500601 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.501298 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.501299 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.502163 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.502166 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.502606 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.502607 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.502704 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.503271 (+ 567us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:18.503281 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:18.504695 (+ 1414us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:18.506056 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.506062 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.506064 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.506355 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.506519 (+ 164us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.506520 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.506520 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.506652 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.506858 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:18.506863 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:18.507027 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.507198 (+ 171us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.507371 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":82,"threads_started":1}
I20260923 23:10:18.508060 21151 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43565
I20260923 23:10:18.508152 21151 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:18.508272 21151 heartbeater.cc:511] Master 127.16.122.190:43565 requested a full tablet report, sending...
I20260923 23:10:18.508749 20961 ts_manager.cc:194] Registered new tserver with Master: fe45358b4f664370b64fdb0ae635b873 (127.16.122.129:42565)
I20260923 23:10:18.509398 20961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:36459
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:18.521848 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:18.604264 21156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:18.604475 21156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:18.604498 21156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:18.605616 21156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:18.605654 21156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:18.605679 21156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:18.607201 21156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:18.607434 21156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:18.607633 21156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:18.612360 21156 server_base.cc:1034] running on GCE node
W20260923 23:10:18.612335 21161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.612393 21164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.612342 21162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:18.612712 21156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:18.612968 21156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:18.614130 21156 hybrid_clock.cc:648] HybridClock initialized: now 1790205018614113 us; error 29 us; skew 500 ppm
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:18.616655 21156 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:18.616981 21156 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:18.617031 21156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:18.617152 21156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:18.618185 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "fac2d25ac4e345098291323066da39dc"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "368e2d209570802376c988acca358fa1"
server_key_iv: "2b1909f9e1d6b5fc4f91038b36148aa9"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.618512 21156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "fac2d25ac4e345098291323066da39dc"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "368e2d209570802376c988acca358fa1"
server_key_iv: "2b1909f9e1d6b5fc4f91038b36148aa9"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.619799 21156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:18.620471 21170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.620610 21156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:18.620690 21156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "fac2d25ac4e345098291323066da39dc"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "368e2d209570802376c988acca358fa1"
server_key_iv: "2b1909f9e1d6b5fc4f91038b36148aa9"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.620754 21156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:18.644191 21156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:18.646322 21156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:18.646468 21156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:18.646688 21156 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:18.646986 21156 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:18.647019 21156 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.647090 21156 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:18.647116 21156 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.652606 21156 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:41077
I20260923 23:10:18.652648 21283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:41077 every 8 connection(s)
I20260923 23:10:18.652961 21156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:18.656296 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21156
I20260923 23:10:18.656400 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:18.656581 16874 external_mini_cluster.cc:1428] Setting key 1ca4070abf5aaa095ce3a286e01fa58b
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:18.662273 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.653619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:43899 (local address 127.16.122.190:43565)
0923 23:10:18.653676 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.653678 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.653869 (+ 191us) server_negotiation.cc:407] Connection header received
0923 23:10:18.654329 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.654330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.654339 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.654423 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:18.655375 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.655872 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.657118 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.657362 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.657435 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.659651 (+ 2216us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:18.659652 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:18.659653 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:18.659660 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:18.661047 (+ 1387us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.661405 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.661406 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.661406 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.661467 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.661671 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.661671 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.661672 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.661807 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:18.661875 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.662097 (+ 222us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.662177 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:18.662346 21286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.653581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43565 (local address 127.16.122.130:43899)
0923 23:10:18.653745 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.653787 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.654253 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.654493 (+ 240us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.654496 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.654755 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:18.655254 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.655259 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.655941 (+ 682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.655942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.657038 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.657041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.657456 (+ 415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.657457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.657608 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.658091 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:18.658100 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:18.659565 (+ 1465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:18.661123 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.661129 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.661130 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.661336 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.661526 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.661526 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.661526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.661616 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.661863 (+ 247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:18.661866 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:18.661977 (+ 111us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.662060 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.662176 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260923 23:10:18.662808 21284 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43565
I20260923 23:10:18.662904 21284 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:18.663095 21284 heartbeater.cc:511] Master 127.16.122.190:43565 requested a full tablet report, sending...
I20260923 23:10:18.663447 20961 ts_manager.cc:194] Registered new tserver with Master: fac2d25ac4e345098291323066da39dc (127.16.122.130:41077)
I20260923 23:10:18.663938 20961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:43899
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:18.680207 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:38979
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:18.758859 21289 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:18.759085 21289 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:18.759109 21289 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:18.760214 21289 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:18.760245 21289 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:18.760270 21289 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:18.761693 21289 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:38979
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43565
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21289
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:18.761873 21289 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:18.762048 21289 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:18.766701 21295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.766712 21297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:18.766716 21294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:18.766780 21289 server_base.cc:1034] running on GCE node
I20260923 23:10:18.767136 21289 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:18.767378 21289 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:18.768548 21289 hybrid_clock.cc:648] HybridClock initialized: now 1790205018768533 us; error 30 us; skew 500 ppm
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:18.770869 21289 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:18.771217 21289 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:18.771270 21289 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:18.771404 21289 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:18.772478 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "c2983d8ba9644c70b799fc7433488802"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "aec728583646dbae0290ed2530a77628"
server_key_iv: "1cdbcbdfd781e83e2a15239aac96e16a"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.772828 21289 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "c2983d8ba9644c70b799fc7433488802"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "aec728583646dbae0290ed2530a77628"
server_key_iv: "1cdbcbdfd781e83e2a15239aac96e16a"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.774070 21289 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:18.774814 21303 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.774977 21289 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:18.775079 21289 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "c2983d8ba9644c70b799fc7433488802"
format_stamp: "Formatted at 2026-09-23 23:10:18 on dist-test-slave-95hh"
server_key: "aec728583646dbae0290ed2530a77628"
server_key_iv: "1cdbcbdfd781e83e2a15239aac96e16a"
server_key_version: "encryptionkey@0"
I20260923 23:10:18.775143 21289 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:18.794842 21289 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:18.796597 21289 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:18.796705 21289 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:18.796896 21289 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:18.797168 21289 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:18.797197 21289 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.797219 21289 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:18.797235 21289 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:18.802785 21289 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39529
I20260923 23:10:18.802819 21416 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39529 every 8 connection(s)
I20260923 23:10:18.803110 21289 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:18.805387 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21289
I20260923 23:10:18.805501 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:18.805696 16874 external_mini_cluster.cc:1428] Setting key 84ed02721c6cf18428bac70f1a8d5c02
Sep 23 23:10:18 dist-test-slave-95hh krb5kdc[20917](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205018, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:18.811451 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.803733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:48707 (local address 127.16.122.190:43565)
0923 23:10:18.803793 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.803795 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.803962 (+ 167us) server_negotiation.cc:407] Connection header received
0923 23:10:18.804419 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.804420 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.804432 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.804480 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:18.805332 (+ 852us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.806050 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.807138 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.807320 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.807371 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.809242 (+ 1871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:18.809243 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:18.809244 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:18.809250 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:18.810376 (+ 1126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.810691 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.810691 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.810692 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.810736 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.810904 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.810904 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.810904 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.810993 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:18.811073 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.811328 (+ 255us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.811390 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:10:18.811609 21419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.803714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43565 (local address 127.16.122.131:48707)
0923 23:10:18.803864 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.803897 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.804343 (+ 446us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.804542 (+ 199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.804545 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.804780 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:18.805253 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.805257 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.806130 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.806131 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.807067 (+ 936us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.807070 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.807378 (+ 308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.807378 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.807471 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.807882 (+ 411us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:18.807889 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:18.809145 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:18.810438 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.810443 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.810445 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.810638 (+ 193us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.810778 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:18.810779 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:18.810779 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:18.810864 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:18.811070 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:18.811075 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:18.811201 (+ 126us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.811301 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.811414 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":86,"threads_started":1}
I20260923 23:10:18.811985 21417 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43565
I20260923 23:10:18.812062 21417 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:18.812172 21417 heartbeater.cc:511] Master 127.16.122.190:43565 requested a full tablet report, sending...
I20260923 23:10:18.812453 20961 ts_manager.cc:194] Registered new tserver with Master: c2983d8ba9644c70b799fc7433488802 (127.16.122.131:39529)
I20260923 23:10:18.812777 20961 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:48707
I20260923 23:10:18.816998 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:10:18.823529 21019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.818192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.16.122.190:43565)
0923 23:10:18.818247 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.818248 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.818335 (+ 87us) server_negotiation.cc:407] Connection header received
0923 23:10:18.818392 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.818393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.818401 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.818446 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:18.818963 (+ 517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.819467 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.820152 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.820332 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.820383 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.821473 (+ 1090us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:18.821474 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:18.821474 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:18.821481 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:18.822446 (+ 965us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.822733 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.822734 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.822735 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.822784 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:18.822941 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:18.822942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:18.822942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:18.823132 (+ 190us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:18.823235 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.823338 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.823408 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:10:18.824999 20961 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:18.826565 20961 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:18.832988 21425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.828534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41077 (local address 127.0.0.1:47236)
0923 23:10:18.828661 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.828699 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.828762 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.829404 (+ 642us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.829405 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.829407 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:18.829456 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.829458 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830723 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.830724 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.831790 (+ 1066us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.831792 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.832696 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.832697 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.832773 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.832787 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.832851 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.832906 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":44,"thread_start_us":74,"threads_started":1}
I20260923 23:10:18.833081 21429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.828917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42565 (local address 127.0.0.1:46898)
0923 23:10:18.829460 (+ 543us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.829468 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.829515 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.829730 (+ 215us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.829730 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.829733 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:18.829770 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.829772 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830906 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.830907 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.831864 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.831867 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.832856 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.832857 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.832909 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.832913 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.832964 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.832995 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":41,"threads_started":1}
I20260923 23:10:18.833216 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.829160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46898 (local address 127.16.122.129:42565)
0923 23:10:18.829583 (+ 423us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.829586 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.829594 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:18.829629 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.829631 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.829644 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.829715 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:18.829855 (+ 140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830829 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.831944 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.832703 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.832769 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.832824 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.833029 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.833103 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":201,"thread_start_us":82,"threads_started":1}
I20260923 23:10:18.833647 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.828740 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39529 (local address 127.0.0.1:51176)
0923 23:10:18.829004 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:18.829013 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:18.829067 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:18.829249 (+ 182us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:18.829252 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:18.829254 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:18.829330 (+ 76us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.829332 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830381 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.830382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.831605 (+ 1223us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:18.831608 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.833407 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.833408 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:18.833480 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.833484 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:18.833544 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:10:18.833583 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":251,"thread_start_us":46,"threads_started":1}
I20260923 23:10:18.833745 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.828655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47236 (local address 127.16.122.130:41077)
0923 23:10:18.829170 (+ 515us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.829173 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.829181 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:18.829218 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.829220 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.829236 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.829350 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:18.829540 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830627 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.831868 (+ 1241us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.832585 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.833522 (+ 937us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.833587 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.833616 (+ 29us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.833660 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":84,"threads_started":1}
I20260923 23:10:18.834010 21085 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba030063f52434e8c102d9b875f0805 (DEFAULT_TABLE table=test-table [id=b4fa7944813341b089b38a7a655b89cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:18.834323 21085 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba030063f52434e8c102d9b875f0805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:18.834301 21218 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba030063f52434e8c102d9b875f0805 (DEFAULT_TABLE table=test-table [id=b4fa7944813341b089b38a7a655b89cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:18.834527 21218 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba030063f52434e8c102d9b875f0805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:18.837007 21431 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Bootstrap starting.
I20260923 23:10:18.837039 21432 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Bootstrap starting.
I20260923 23:10:18.837455 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:18.828845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51176 (local address 127.16.122.131:39529)
0923 23:10:18.828998 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:18.829011 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:18.829057 (+ 46us) server_negotiation.cc:407] Connection header received
0923 23:10:18.829126 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:18.829128 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:18.829145 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:18.829228 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:18.829425 (+ 197us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.830260 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.831684 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:18.832273 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:18.832334 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:18.832388 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:18.837305 (+ 4917us) server_negotiation.cc:299] Negotiation successful
0923 23:10:18.837372 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":83,"threads_started":1}
I20260923 23:10:18.837531 21431 tablet_bootstrap.cc:654] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:18.837653 21432 tablet_bootstrap.cc:654] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:18.837997 21431 log.cc:824] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:18.838064 21351 tablet_service.cc:1467] Processing CreateTablet for tablet 6ba030063f52434e8c102d9b875f0805 (DEFAULT_TABLE table=test-table [id=b4fa7944813341b089b38a7a655b89cd]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:18.838395 21432 log.cc:824] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:18.838414 21351 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 6ba030063f52434e8c102d9b875f0805. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:18.838683 21431 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: No bootstrap required, opened a new log
I20260923 23:10:18.838753 21431 ts_tablet_manager.cc:1397] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:18.839149 21432 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: No bootstrap required, opened a new log
I20260923 23:10:18.839223 21432 ts_tablet_manager.cc:1397] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:18.840332 21431 raft_consensus.cc:348] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.840457 21431 raft_consensus.cc:374] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:18.840487 21431 raft_consensus.cc:729] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe45358b4f664370b64fdb0ae635b873, State: Initialized, Role: FOLLOWER
I20260923 23:10:18.840622 21431 consensus_queue.cc:260] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.840867 21431 ts_tablet_manager.cc:1428] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:18.840984 21151 heartbeater.cc:503] Master 127.16.122.190:43565 was elected leader, sending a full tablet report...
I20260923 23:10:18.841171 21432 raft_consensus.cc:348] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.841341 21432 raft_consensus.cc:374] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:18.841387 21432 raft_consensus.cc:729] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fac2d25ac4e345098291323066da39dc, State: Initialized, Role: FOLLOWER
I20260923 23:10:18.841506 21432 consensus_queue.cc:260] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.841714 21436 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Bootstrap starting.
I20260923 23:10:18.841786 21432 ts_tablet_manager.cc:1428] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:10:18.841967 21284 heartbeater.cc:503] Master 127.16.122.190:43565 was elected leader, sending a full tablet report...
I20260923 23:10:18.842428 21436 tablet_bootstrap.cc:654] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:18.842952 21436 log.cc:824] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:18.843627 21436 tablet_bootstrap.cc:492] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: No bootstrap required, opened a new log
I20260923 23:10:18.843677 21436 ts_tablet_manager.cc:1397] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:18.845115 21436 raft_consensus.cc:348] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.845239 21436 raft_consensus.cc:374] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:18.845258 21436 raft_consensus.cc:729] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c2983d8ba9644c70b799fc7433488802, State: Initialized, Role: FOLLOWER
I20260923 23:10:18.845340 21436 consensus_queue.cc:260] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:18.845559 21436 ts_tablet_manager.cc:1428] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:18.845608 21417 heartbeater.cc:503] Master 127.16.122.190:43565 was elected leader, sending a full tablet report...
W20260923 23:10:18.903456 21285 tablet.cc:2369] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:18.998581 21152 tablet.cc:2369] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:19.053642 21418 tablet.cc:2369] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:19.066354 21435 raft_consensus.cc:482] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:19.066481 21435 raft_consensus.cc:504] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:19.066803 21435 leader_election.cc:288] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fac2d25ac4e345098291323066da39dc (127.16.122.130:41077), c2983d8ba9644c70b799fc7433488802 (127.16.122.131:39529)
I20260923 23:10:19.069694 21428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.067317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:47961 (local address 127.16.122.131:39529)
0923 23:10:19.067381 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.067383 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.067391 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:19.067455 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.067456 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.067465 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.067529 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:19.067665 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.068097 (+ 432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069018 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.069395 (+ 377us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069459 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.069466 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.069598 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.069629 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:10:19.069700 21426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.067139 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50601 (local address 127.16.122.130:41077)
0923 23:10:19.067198 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.067200 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.067212 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:19.067392 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.067393 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.067402 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.067455 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:19.067592 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.068084 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069020 (+ 936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.069383 (+ 363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069436 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.069444 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.069579 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.069616 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:10:19.069717 21440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.066995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41077 (local address 127.16.122.129:50601)
0923 23:10:19.067156 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.067168 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.067223 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.067467 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.067467 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.067470 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:19.067528 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.067530 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.068158 (+ 628us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.068159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.068929 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.068931 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.069456 (+ 525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.069516 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.069520 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.069559 (+ 39us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.069598 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":104,"threads_started":1}
I20260923 23:10:19.069725 21441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.067215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39529 (local address 127.16.122.129:47961)
0923 23:10:19.067308 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.067319 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.067388 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.067537 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.067538 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.067540 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:19.067607 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.067609 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.068165 (+ 556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.068166 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.068936 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.068937 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.069466 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.069467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.069534 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.069538 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.069588 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.069623 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78,"mutex_wait_us":10,"thread_start_us":39,"threads_started":1}
I20260923 23:10:19.070052 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba030063f52434e8c102d9b875f0805" candidate_uuid: "fe45358b4f664370b64fdb0ae635b873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac2d25ac4e345098291323066da39dc" is_pre_election: true
I20260923 23:10:19.070214 21238 raft_consensus.cc:2454] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe45358b4f664370b64fdb0ae635b873 in term 0.
I20260923 23:10:19.070283 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba030063f52434e8c102d9b875f0805" candidate_uuid: "fe45358b4f664370b64fdb0ae635b873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2983d8ba9644c70b799fc7433488802" is_pre_election: true
I20260923 23:10:19.070413 21040 leader_election.cc:302] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: fac2d25ac4e345098291323066da39dc, fe45358b4f664370b64fdb0ae635b873; no voters:
I20260923 23:10:19.070438 21371 raft_consensus.cc:2454] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fe45358b4f664370b64fdb0ae635b873 in term 0.
I20260923 23:10:19.070545 21435 raft_consensus.cc:2781] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:19.070593 21435 raft_consensus.cc:482] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:19.070614 21435 raft_consensus.cc:3037] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.071309 21435 raft_consensus.cc:504] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:19.071419 21435 leader_election.cc:288] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [CANDIDATE]: Term 1 election: Requested vote from peers fac2d25ac4e345098291323066da39dc (127.16.122.130:41077), c2983d8ba9644c70b799fc7433488802 (127.16.122.131:39529)
I20260923 23:10:19.071595 21371 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba030063f52434e8c102d9b875f0805" candidate_uuid: "fe45358b4f664370b64fdb0ae635b873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c2983d8ba9644c70b799fc7433488802"
I20260923 23:10:19.071606 21238 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "6ba030063f52434e8c102d9b875f0805" candidate_uuid: "fe45358b4f664370b64fdb0ae635b873" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fac2d25ac4e345098291323066da39dc"
I20260923 23:10:19.071671 21238 raft_consensus.cc:3037] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.071671 21371 raft_consensus.cc:3037] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.072342 21238 raft_consensus.cc:2454] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe45358b4f664370b64fdb0ae635b873 in term 1.
I20260923 23:10:19.072364 21371 raft_consensus.cc:2454] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fe45358b4f664370b64fdb0ae635b873 in term 1.
I20260923 23:10:19.072492 21040 leader_election.cc:302] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c2983d8ba9644c70b799fc7433488802, fe45358b4f664370b64fdb0ae635b873; no voters:
I20260923 23:10:19.072608 21435 raft_consensus.cc:2781] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:19.072777 21435 raft_consensus.cc:686] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [term 1 LEADER]: Becoming Leader. State: Replica: fe45358b4f664370b64fdb0ae635b873, State: Running, Role: LEADER
I20260923 23:10:19.072875 21435 consensus_queue.cc:237] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } }
I20260923 23:10:19.073722 20961 catalog_manager.cc:5495] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 reported cstate change: term changed from 0 to 1, leader changed from <none> to fe45358b4f664370b64fdb0ae635b873 (127.16.122.129). New cstate: current_term: 1 leader_uuid: "fe45358b4f664370b64fdb0ae635b873" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fe45358b4f664370b64fdb0ae635b873" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42565 } health_report { overall_health: HEALTHY } } }
I20260923 23:10:19.113881 21430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.111692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46914 (local address 127.16.122.129:42565)
0923 23:10:19.111753 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.111755 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.111762 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:19.111798 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.111799 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.111816 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.111875 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:19.112009 (+ 134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.112462 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.113185 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.113365 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.113425 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.113571 (+ 146us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:19.113673 (+ 102us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:19.113745 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.113782 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.113803 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:10:19.115698 21238 raft_consensus.cc:1261] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 1 FOLLOWER]: Refusing update from remote peer fe45358b4f664370b64fdb0ae635b873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:19.115698 21371 raft_consensus.cc:1261] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 1 FOLLOWER]: Refusing update from remote peer fe45358b4f664370b64fdb0ae635b873: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:19.116066 21442 consensus_queue.cc:1035] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fac2d25ac4e345098291323066da39dc" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41077 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:19.116209 21435 consensus_queue.cc:1035] T 6ba030063f52434e8c102d9b875f0805 P fe45358b4f664370b64fdb0ae635b873 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c2983d8ba9644c70b799fc7433488802" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39529 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:19.118641 21446 mvcc.cc:205] Tried to move back new op lower bound from 7332679758295040000 to 7332679758123106304. Current Snapshot: MvccSnapshot[applied={T|T < 7332679758295040000}]
I20260923 23:10:19.120878 20960 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:19.120971 20960 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51506:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:19.121708 20960 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a33d8e59f17b4c998d172a7e6a35b04f: Sending DeleteTablet for 3 replicas of tablet 6ba030063f52434e8c102d9b875f0805
I20260923 23:10:19.122059 21085 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba030063f52434e8c102d9b875f0805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:46898
I20260923 23:10:19.122054 21351 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba030063f52434e8c102d9b875f0805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:51176
I20260923 23:10:19.122087 21218 tablet_service.cc:1514] Processing DeleteTablet for tablet 6ba030063f52434e8c102d9b875f0805 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47236
I20260923 23:10:19.122298 21455 tablet_replica.cc:330] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: stopping tablet replica
I20260923 23:10:19.122399 21455 raft_consensus.cc:2229] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:19.122498 21455 raft_consensus.cc:2258] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:19.122591 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21023
I20260923 23:10:19.123209 21457 tablet_replica.cc:330] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: stopping tablet replica
I20260923 23:10:19.123307 21457 raft_consensus.cc:2229] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:19.123378 21457 raft_consensus.cc:2258] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:19.123739 21457 ts_tablet_manager.cc:1905] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:19.124742 21455 ts_tablet_manager.cc:1905] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:19.125214 21457 ts_tablet_manager.cc:1918] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:19.125283 21457 log.cc:1196] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/6ba030063f52434e8c102d9b875f0805
I20260923 23:10:19.125520 21457 ts_tablet_manager.cc:1939] T 6ba030063f52434e8c102d9b875f0805 P fac2d25ac4e345098291323066da39dc: Deleting consensus metadata
I20260923 23:10:19.125926 20947 catalog_manager.cc:4841] TS fac2d25ac4e345098291323066da39dc (127.16.122.130:41077): tablet 6ba030063f52434e8c102d9b875f0805 (table test-table [id=b4fa7944813341b089b38a7a655b89cd]) successfully deleted
I20260923 23:10:19.126175 21455 ts_tablet_manager.cc:1918] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:19.126230 21455 log.cc:1196] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/6ba030063f52434e8c102d9b875f0805
I20260923 23:10:19.126398 21455 ts_tablet_manager.cc:1939] T 6ba030063f52434e8c102d9b875f0805 P c2983d8ba9644c70b799fc7433488802: Deleting consensus metadata
I20260923 23:10:19.126792 20947 catalog_manager.cc:4841] TS c2983d8ba9644c70b799fc7433488802 (127.16.122.131:39529): tablet 6ba030063f52434e8c102d9b875f0805 (table test-table [id=b4fa7944813341b089b38a7a655b89cd]) successfully deleted
W20260923 23:10:19.127847 20947 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:42565 (error 108)
I20260923 23:10:19.128085 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21156
I20260923 23:10:19.128284 21427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.128182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42565 (local address 127.0.0.1:46922)
0923 23:10:19.128226 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.128256 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42565: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:10:19.128384 20947 catalog_manager.cc:4584] TS fe45358b4f664370b64fdb0ae635b873 (127.16.122.129:42565): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 6ba030063f52434e8c102d9b875f0805: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42565: connect: Connection refused (error 111)
I20260923 23:10:19.132915 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21289
I20260923 23:10:19.136690 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 20930
2026-09-23T23:10:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (1982 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-09-23T23:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:19Z Disabled control of system clock
I20260923 23:10:19.161481 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34305
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34305
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:10:19.238065 21462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:19.238276 21462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:19.238304 21462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:19.239476 21462 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:19.239518 21462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:19.239537 21462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:19.239552 21462 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:19.239568 21462 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:19.240916 21462 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34305
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34305
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:19.241153 21462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:19.241412 21462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:19.245824 21468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.245927 21470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.246001 21462 server_base.cc:1034] running on GCE node
W20260923 23:10:19.245867 21467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.246312 21462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:19.246593 21462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:19.247723 21462 hybrid_clock.cc:648] HybridClock initialized: now 1790205019247712 us; error 31 us; skew 500 ppm
I20260923 23:10:19.248869 21462 webserver.cc:460] Webserver started at http://127.16.122.190:43309/ using document root <none> and password file <none>
I20260923 23:10:19.249089 21462 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:19.249135 21462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:19.249256 21462 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:19.250273 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "e282bb0bcf714ebaafc8f3ac6edc9453"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "836056264778ed4bfe07b2c01bf57db0"
server_key_iv: "18102d6dd92fbc35ee76215879cfa1ac"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.250589 21462 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "e282bb0bcf714ebaafc8f3ac6edc9453"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "836056264778ed4bfe07b2c01bf57db0"
server_key_iv: "18102d6dd92fbc35ee76215879cfa1ac"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.251812 21462 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:19.252535 21476 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.252693 21462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:19.252780 21462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "e282bb0bcf714ebaafc8f3ac6edc9453"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "836056264778ed4bfe07b2c01bf57db0"
server_key_iv: "18102d6dd92fbc35ee76215879cfa1ac"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.252842 21462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:19.256060 21462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:19.256287 21462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:19.256395 21462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:19.259861 21462 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34305
I20260923 23:10:19.259895 21528 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34305 every 8 connection(s)
I20260923 23:10:19.260192 21462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:19.260787 21529 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:19.263461 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: Bootstrap starting.
I20260923 23:10:19.263988 21529 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:19.264241 21529 log.cc:824] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:19.264766 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21462
I20260923 23:10:19.264843 21529 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: No bootstrap required, opened a new log
I20260923 23:10:19.264858 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:19.265035 16874 external_mini_cluster.cc:1428] Setting key a94a7c0c6d52c761d42d98ea31df579a
I20260923 23:10:19.266973 21529 raft_consensus.cc:348] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } }
I20260923 23:10:19.267181 21529 raft_consensus.cc:374] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:19.267241 21529 raft_consensus.cc:729] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e282bb0bcf714ebaafc8f3ac6edc9453, State: Initialized, Role: FOLLOWER
I20260923 23:10:19.267381 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.265513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33190 (local address 127.16.122.190:34305)
0923 23:10:19.265680 (+ 167us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.265687 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.265710 (+ 23us) server_negotiation.cc:407] Connection header received
0923 23:10:19.266122 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.266128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.266164 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.266273 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.266887 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.266891 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.266891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.266894 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.266934 (+ 40us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.266944 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.266987 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.267115 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.267168 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":59,"threads_started":1}
I20260923 23:10:19.267371 21529 consensus_queue.cc:260] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } }
I20260923 23:10:19.267452 21529 raft_consensus.cc:388] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:19.267485 21529 raft_consensus.cc:482] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:19.267534 21529 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.268309 21529 raft_consensus.cc:504] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } }
I20260923 23:10:19.268404 21529 leader_election.cc:302] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e282bb0bcf714ebaafc8f3ac6edc9453; no voters:
I20260923 23:10:19.268544 21529 leader_election.cc:288] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:19.268570 21534 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:19.268675 21534 raft_consensus.cc:686] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [term 1 LEADER]: Becoming Leader. State: Replica: e282bb0bcf714ebaafc8f3ac6edc9453, State: Running, Role: LEADER
I20260923 23:10:19.268762 21534 consensus_queue.cc:237] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } }
I20260923 23:10:19.268781 21529 sys_catalog.cc:564] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:19.269117 21535 sys_catalog.cc:455] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } } }
I20260923 23:10:19.269100 21536 sys_catalog.cc:455] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e282bb0bcf714ebaafc8f3ac6edc9453. Latest consensus state: current_term: 1 leader_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e282bb0bcf714ebaafc8f3ac6edc9453" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34305 } } }
I20260923 23:10:19.269214 21536 sys_catalog.cc:458] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:19.269210 21535 sys_catalog.cc:458] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453 [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:19.270483 21550 catalog_manager.cc:1558] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:19.270545 21550 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:19.270589 21541 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:19.270747 21541 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:19.271999 21541 catalog_manager.cc:1347] Generated new cluster ID: c80eccbc08964bc48c88a27d6664d6f8
I20260923 23:10:19.272049 21541 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:19.282266 21541 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:19.282461 21541 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:19.288821 21541 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: Generated new TSK 0
I20260923 23:10:19.289000 21541 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:19.291008 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--builtin_ntp_servers=127.16.122.148:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:10:19.386096 21553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:19.386308 21553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:19.386339 21553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:19.387574 21553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:19.387616 21553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:19.387650 21553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:19.388996 21553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21553
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:19.389236 21553 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:19.389524 21553 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:19.394595 21558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.394696 21559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.394604 21561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.395231 21553 server_base.cc:1034] running on GCE node
I20260923 23:10:19.395468 21553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:19.395722 21553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:19.396955 21553 hybrid_clock.cc:648] HybridClock initialized: now 1790205019396935 us; error 34 us; skew 500 ppm
I20260923 23:10:19.398227 21553 webserver.cc:460] Webserver started at http://127.16.122.129:35115/ using document root <none> and password file <none>
I20260923 23:10:19.398476 21553 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:19.398528 21553 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:19.398654 21553 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:19.399863 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "35b82a19a8bf44e18280b5e36074663c"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "2b2081b7e3c16a142d45935aa9133b9e"
server_key_iv: "a32e345f1515b7401c98878deaad636f"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.400188 21553 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "35b82a19a8bf44e18280b5e36074663c"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "2b2081b7e3c16a142d45935aa9133b9e"
server_key_iv: "a32e345f1515b7401c98878deaad636f"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.401502 21553 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:19.402196 21567 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.402356 21553 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:19.402462 21553 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "35b82a19a8bf44e18280b5e36074663c"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "2b2081b7e3c16a142d45935aa9133b9e"
server_key_iv: "a32e345f1515b7401c98878deaad636f"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.402525 21553 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:19.405794 21553 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:19.406045 21553 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:19.406162 21553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:19.406350 21553 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:19.406632 21553 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:19.406662 21553 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.406689 21553 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:19.406713 21553 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.412089 21553 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:46541
I20260923 23:10:19.412156 21680 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:46541 every 8 connection(s)
I20260923 23:10:19.412436 21553 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:19.415252 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.413097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:42757 (local address 127.16.122.190:34305)
0923 23:10:19.413147 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.413149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.413379 (+ 230us) server_negotiation.cc:407] Connection header received
0923 23:10:19.413868 (+ 489us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.413869 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.413876 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.413947 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.414985 (+ 1038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.414987 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.414987 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.414991 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.415007 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.415011 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.415098 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.415171 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.415186 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:19.415439 21683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.413097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34305 (local address 127.16.122.129:42757)
0923 23:10:19.413265 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.413299 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.413775 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.414019 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.414023 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.414031 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.414847 (+ 816us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.414848 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.414874 (+ 26us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.414901 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.415102 (+ 201us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.415107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.415111 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.415176 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.415236 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":71,"threads_started":1}
I20260923 23:10:19.415899 21681 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34305
I20260923 23:10:19.415990 21681 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:19.416023 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21553
I20260923 23:10:19.416119 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:19.416141 21681 heartbeater.cc:511] Master 127.16.122.190:34305 requested a full tablet report, sending...
I20260923 23:10:19.416280 16874 external_mini_cluster.cc:1428] Setting key 010aab9dc9eb403e076fb970833911b4
I20260923 23:10:19.416649 21493 ts_manager.cc:194] Registered new tserver with Master: 35b82a19a8bf44e18280b5e36074663c (127.16.122.129:46541)
I20260923 23:10:19.417153 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--builtin_ntp_servers=127.16.122.148:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:10:19.498533 21684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:19.498744 21684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:19.498767 21684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:19.500041 21684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:19.500085 21684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:19.500113 21684 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:19.501493 21684 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21684
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:19.501732 21684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:19.502029 21684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:19.506841 21692 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.506855 21689 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.506893 21690 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.507259 21684 server_base.cc:1034] running on GCE node
I20260923 23:10:19.507511 21684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:19.507730 21684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:19.508881 21684 hybrid_clock.cc:648] HybridClock initialized: now 1790205019508866 us; error 29 us; skew 500 ppm
I20260923 23:10:19.510149 21684 webserver.cc:460] Webserver started at http://127.16.122.130:39319/ using document root <none> and password file <none>
I20260923 23:10:19.510401 21684 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:19.510447 21684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:19.510568 21684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:19.511669 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "060ee373d5964855acdce98ab1b52197"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "905f33de4ac5f600602b73143b9318bf"
server_key_iv: "ac9406acd4265d310b374a8810e81491"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.511981 21684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "060ee373d5964855acdce98ab1b52197"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "905f33de4ac5f600602b73143b9318bf"
server_key_iv: "ac9406acd4265d310b374a8810e81491"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.513526 21684 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:19.514169 21698 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.514292 21684 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:19.514396 21684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "060ee373d5964855acdce98ab1b52197"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "905f33de4ac5f600602b73143b9318bf"
server_key_iv: "ac9406acd4265d310b374a8810e81491"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.514463 21684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:19.517772 21684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:19.518030 21684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:19.518148 21684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:19.518343 21684 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:19.518623 21684 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:19.518653 21684 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.518680 21684 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:19.518702 21684 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.524153 21684 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:38131
I20260923 23:10:19.524195 21811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:38131 every 8 connection(s)
I20260923 23:10:19.524520 21684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:19.527351 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.525234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:36267 (local address 127.16.122.190:34305)
0923 23:10:19.525309 (+ 75us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.525311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.525519 (+ 208us) server_negotiation.cc:407] Connection header received
0923 23:10:19.525974 (+ 455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.525975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.525980 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.526045 (+ 65us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.527145 (+ 1100us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.527146 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.527146 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.527150 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.527165 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.527169 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.527213 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.527281 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.527292 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20260923 23:10:19.527529 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.525187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34305 (local address 127.16.122.130:36267)
0923 23:10:19.525386 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.525434 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.525871 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.526110 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.526114 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.526122 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.526947 (+ 825us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.526948 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.526976 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.527005 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.527218 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.527222 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.527224 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.527277 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.527328 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":85,"threads_started":1}
I20260923 23:10:19.527980 21812 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34305
I20260923 23:10:19.528057 21812 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:19.528193 21812 heartbeater.cc:511] Master 127.16.122.190:34305 requested a full tablet report, sending...
I20260923 23:10:19.528518 21493 ts_manager.cc:194] Registered new tserver with Master: 060ee373d5964855acdce98ab1b52197 (127.16.122.130:38131)
I20260923 23:10:19.531234 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21684
I20260923 23:10:19.531317 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:19.531476 16874 external_mini_cluster.cc:1428] Setting key ba7519f460efdc2a4a01593e11b93295
I20260923 23:10:19.532496 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--builtin_ntp_servers=127.16.122.148:41143
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260923 23:10:19.611112 21815 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:19.611321 21815 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:19.611342 21815 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:19.612514 21815 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:19.612555 21815 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:19.612581 21815 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:19.613933 21815 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41143
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34305
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21815
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:19.614169 21815 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:19.614457 21815 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:19.619280 21820 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.619355 21815 server_base.cc:1034] running on GCE node
W20260923 23:10:19.619252 21823 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.619252 21821 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.619647 21815 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:19.619864 21815 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:19.621011 21815 hybrid_clock.cc:648] HybridClock initialized: now 1790205019620998 us; error 29 us; skew 500 ppm
I20260923 23:10:19.622184 21815 webserver.cc:460] Webserver started at http://127.16.122.131:33127/ using document root <none> and password file <none>
I20260923 23:10:19.622398 21815 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:19.622447 21815 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:19.622570 21815 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:19.623649 21815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "c92ecd98f107497db3daae5069db4cce"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "9c673f0f42374aed14685d9a5e808bd8"
server_key_iv: "235977c07a10849c016aa6b5f99854a2"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.623976 21815 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "c92ecd98f107497db3daae5069db4cce"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "9c673f0f42374aed14685d9a5e808bd8"
server_key_iv: "235977c07a10849c016aa6b5f99854a2"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.625172 21815 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:19.625875 21829 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.626048 21815 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:19.626125 21815 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "c92ecd98f107497db3daae5069db4cce"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "9c673f0f42374aed14685d9a5e808bd8"
server_key_iv: "235977c07a10849c016aa6b5f99854a2"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.626185 21815 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:19.629521 21815 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:19.629765 21815 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:19.629882 21815 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:19.630072 21815 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:19.630350 21815 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:19.630381 21815 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.630407 21815 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:19.630430 21815 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.635676 21815 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:46301
I20260923 23:10:19.635749 21942 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:46301 every 8 connection(s)
I20260923 23:10:19.636029 21815 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:19.636736 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21815
I20260923 23:10:19.636852 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:19.637090 16874 external_mini_cluster.cc:1428] Setting key b64d1525681d60c73e4277b074aaa1f2
I20260923 23:10:19.638983 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.636751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:51137 (local address 127.16.122.190:34305)
0923 23:10:19.636815 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.636818 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.637020 (+ 202us) server_negotiation.cc:407] Connection header received
0923 23:10:19.637544 (+ 524us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.637545 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.637551 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.637621 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.638782 (+ 1161us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.638783 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.638783 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.638787 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.638803 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.638807 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.638867 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.638915 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.638926 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:19.639196 21945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.636708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34305 (local address 127.16.122.131:51137)
0923 23:10:19.636900 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.636939 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.637448 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.637696 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.637700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.637710 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.638631 (+ 921us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.638632 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.638660 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.638689 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.638869 (+ 180us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.638872 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.638875 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.638922 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.638970 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
I20260923 23:10:19.639672 21943 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34305
I20260923 23:10:19.639756 21943 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:19.639914 21943 heartbeater.cc:511] Master 127.16.122.190:34305 requested a full tablet report, sending...
I20260923 23:10:19.640269 21493 ts_manager.cc:194] Registered new tserver with Master: c92ecd98f107497db3daae5069db4cce (127.16.122.131:46301)
I20260923 23:10:19.648805 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:10:19.655611 21533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.654391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33202 (local address 127.16.122.190:34305)
0923 23:10:19.654441 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.654444 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.654574 (+ 130us) server_negotiation.cc:407] Connection header received
0923 23:10:19.654638 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.654639 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.654645 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.654699 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.655344 (+ 645us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.655345 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.655346 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.655367 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.655382 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.655387 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.655435 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.655505 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.655521 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:19.657370 21493 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33202:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:19.658903 21493 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:19.662627 21952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:38131 (local address 127.0.0.1:53388)
0923 23:10:19.661544 (+ 412us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.661553 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.661610 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.661751 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.661755 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.661761 (+ 6us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.662247 (+ 486us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.662247 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.662263 (+ 16us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.662287 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.662476 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.662482 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.662485 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.662543 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.662555 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"thread_start_us":119,"threads_started":1}
I20260923 23:10:19.662858 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53388 (local address 127.16.122.130:38131)
0923 23:10:19.661493 (+ 152us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.661497 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.661632 (+ 135us) server_negotiation.cc:407] Connection header received
0923 23:10:19.661662 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.661664 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.661673 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.661743 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.662357 (+ 614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.662365 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.662365 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.662370 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.662404 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.662425 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.662475 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.662732 (+ 257us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.662750 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":66,"threads_started":1}
I20260923 23:10:19.663146 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:46541 (local address 127.0.0.1:49816)
0923 23:10:19.661543 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.661553 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.661610 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.662116 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.662117 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.662119 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.662534 (+ 415us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.662535 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.662540 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.662551 (+ 11us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.663006 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.663007 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.663007 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.663071 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.663084 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":396,"spinlock_wait_cycles":896,"thread_start_us":149,"threads_started":1}
I20260923 23:10:19.663226 21953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:46301 (local address 127.0.0.1:45028)
0923 23:10:19.661333 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.661375 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.661463 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.662222 (+ 759us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.662223 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.662226 (+ 3us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.662806 (+ 580us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.662806 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.662812 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.662822 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.663070 (+ 248us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.663071 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.663071 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.663110 (+ 39us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.663122 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172,"thread_start_us":62,"threads_started":1}
I20260923 23:10:19.663269 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45028 (local address 127.16.122.131:46301)
0923 23:10:19.661544 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.661548 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.661557 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:19.661580 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.661582 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.661589 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.661656 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.662895 (+ 1239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.662904 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.662905 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.662909 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.662943 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.662970 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.663021 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.663178 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.663194 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165,"thread_start_us":74,"threads_started":1}
I20260923 23:10:19.663391 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.661304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49816 (local address 127.16.122.129:46541)
0923 23:10:19.661484 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.661489 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.661996 (+ 507us) server_negotiation.cc:407] Connection header received
0923 23:10:19.662026 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.662028 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.662038 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.662116 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.662610 (+ 494us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.662618 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.662619 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.662622 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.662657 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.662916 (+ 259us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.662980 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.663297 (+ 317us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.663314 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":74,"threads_started":1}
I20260923 23:10:19.663600 21746 tablet_service.cc:1467] Processing CreateTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c (DEFAULT_TABLE table=test-table [id=57c5ebb9ba5644a0bd5a273eb491f1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:19.663951 21746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e622a8220ad4e37b464abb41b9ed87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:19.663857 21877 tablet_service.cc:1467] Processing CreateTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c (DEFAULT_TABLE table=test-table [id=57c5ebb9ba5644a0bd5a273eb491f1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:19.664024 21615 tablet_service.cc:1467] Processing CreateTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c (DEFAULT_TABLE table=test-table [id=57c5ebb9ba5644a0bd5a273eb491f1cf]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:19.664148 21877 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e622a8220ad4e37b464abb41b9ed87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:19.664294 21615 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4e622a8220ad4e37b464abb41b9ed87c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:19.666471 21957 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Bootstrap starting.
I20260923 23:10:19.667013 21957 tablet_bootstrap.cc:654] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:19.667143 21958 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Bootstrap starting.
I20260923 23:10:19.667357 21957 log.cc:824] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:19.667403 21959 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: Bootstrap starting.
I20260923 23:10:19.668131 21959 tablet_bootstrap.cc:654] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:19.668426 21959 log.cc:824] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:19.668396 21957 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: No bootstrap required, opened a new log
I20260923 23:10:19.668620 21957 ts_tablet_manager.cc:1397] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:19.668771 21958 tablet_bootstrap.cc:654] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:19.669205 21958 log.cc:824] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:19.669351 21959 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: No bootstrap required, opened a new log
I20260923 23:10:19.669416 21959 ts_tablet_manager.cc:1397] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:19.670008 21958 tablet_bootstrap.cc:492] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: No bootstrap required, opened a new log
I20260923 23:10:19.670070 21958 ts_tablet_manager.cc:1397] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.002s
I20260923 23:10:19.671008 21957 raft_consensus.cc:348] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.671202 21957 raft_consensus.cc:374] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:19.671243 21957 raft_consensus.cc:729] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c92ecd98f107497db3daae5069db4cce, State: Initialized, Role: FOLLOWER
I20260923 23:10:19.671293 21959 raft_consensus.cc:348] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.671362 21957 consensus_queue.cc:260] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.671442 21959 raft_consensus.cc:374] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:19.671480 21959 raft_consensus.cc:729] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 35b82a19a8bf44e18280b5e36074663c, State: Initialized, Role: FOLLOWER
I20260923 23:10:19.671588 21959 consensus_queue.cc:260] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.671622 21957 ts_tablet_manager.cc:1428] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 23:10:19.671670 21943 heartbeater.cc:503] Master 127.16.122.190:34305 was elected leader, sending a full tablet report...
I20260923 23:10:19.671823 21959 ts_tablet_manager.cc:1428] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:19.671975 21681 heartbeater.cc:503] Master 127.16.122.190:34305 was elected leader, sending a full tablet report...
I20260923 23:10:19.672475 21958 raft_consensus.cc:348] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.672627 21958 raft_consensus.cc:374] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:19.672660 21958 raft_consensus.cc:729] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 060ee373d5964855acdce98ab1b52197, State: Initialized, Role: FOLLOWER
I20260923 23:10:19.672780 21958 consensus_queue.cc:260] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.673038 21958 ts_tablet_manager.cc:1428] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260923 23:10:19.673058 21812 heartbeater.cc:503] Master 127.16.122.190:34305 was elected leader, sending a full tablet report...
I20260923 23:10:19.725378 21965 raft_consensus.cc:482] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:19.725500 21965 raft_consensus.cc:504] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.725840 21965 leader_election.cc:288] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c92ecd98f107497db3daae5069db4cce (127.16.122.131:46301), 35b82a19a8bf44e18280b5e36074663c (127.16.122.129:46541)
I20260923 23:10:19.726990 21954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.726096 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:59727 (local address 127.16.122.129:46541)
0923 23:10:19.726140 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.726141 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.726153 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:19.726181 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.726182 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.726187 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.726243 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.726758 (+ 515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.726759 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.726760 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.726764 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.726783 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.726787 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.726838 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.726918 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.726930 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:19.727054 21956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.726124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:55423 (local address 127.16.122.131:46301)
0923 23:10:19.726168 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.726170 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.726306 (+ 136us) server_negotiation.cc:407] Connection header received
0923 23:10:19.726329 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.726330 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.726334 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.726401 (+ 67us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.726841 (+ 440us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.726843 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.726843 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.726848 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.726863 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.726867 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.726913 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.726958 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.726970 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:10:19.727128 21966 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.726040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:46301 (local address 127.16.122.130:55423)
0923 23:10:19.726227 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.726235 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.726283 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.726416 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.726417 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.726431 (+ 14us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.726763 (+ 332us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.726763 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.726768 (+ 5us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.726777 (+ 9us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.726916 (+ 139us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.726917 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.726917 (+ 0us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.726958 (+ 41us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.726970 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":173,"mutex_wait_us":13,"thread_start_us":105,"threads_started":1}
I20260923 23:10:19.726990 21814 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.726033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:46541 (local address 127.16.122.130:59727)
0923 23:10:19.726086 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.726096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.726134 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.726266 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.726267 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.726269 (+ 2us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:19.726657 (+ 388us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:19.726658 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:19.726664 (+ 6us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:19.726674 (+ 10us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:19.726862 (+ 188us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:19.726863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:19.726864 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:19.726908 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:10:19.726922 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20260923 23:10:19.727525 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e622a8220ad4e37b464abb41b9ed87c" candidate_uuid: "060ee373d5964855acdce98ab1b52197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ecd98f107497db3daae5069db4cce" is_pre_election: true
I20260923 23:10:19.727538 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e622a8220ad4e37b464abb41b9ed87c" candidate_uuid: "060ee373d5964855acdce98ab1b52197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b82a19a8bf44e18280b5e36074663c" is_pre_election: true
I20260923 23:10:19.727655 21897 raft_consensus.cc:2454] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060ee373d5964855acdce98ab1b52197 in term 0.
I20260923 23:10:19.727663 21635 raft_consensus.cc:2454] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 060ee373d5964855acdce98ab1b52197 in term 0.
I20260923 23:10:19.727869 21700 leader_election.cc:302] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee373d5964855acdce98ab1b52197, c92ecd98f107497db3daae5069db4cce; no voters:
I20260923 23:10:19.727989 21965 raft_consensus.cc:2781] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:19.728039 21965 raft_consensus.cc:482] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:19.728065 21965 raft_consensus.cc:3037] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.728729 21965 raft_consensus.cc:504] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.728837 21965 leader_election.cc:288] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [CANDIDATE]: Term 1 election: Requested vote from peers c92ecd98f107497db3daae5069db4cce (127.16.122.131:46301), 35b82a19a8bf44e18280b5e36074663c (127.16.122.129:46541)
I20260923 23:10:19.728955 21897 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e622a8220ad4e37b464abb41b9ed87c" candidate_uuid: "060ee373d5964855acdce98ab1b52197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c92ecd98f107497db3daae5069db4cce"
I20260923 23:10:19.728999 21635 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4e622a8220ad4e37b464abb41b9ed87c" candidate_uuid: "060ee373d5964855acdce98ab1b52197" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "35b82a19a8bf44e18280b5e36074663c"
I20260923 23:10:19.729034 21897 raft_consensus.cc:3037] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.729051 21635 raft_consensus.cc:3037] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.729688 21635 raft_consensus.cc:2454] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060ee373d5964855acdce98ab1b52197 in term 1.
I20260923 23:10:19.729830 21699 leader_election.cc:302] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 060ee373d5964855acdce98ab1b52197, 35b82a19a8bf44e18280b5e36074663c; no voters:
I20260923 23:10:19.729851 21897 raft_consensus.cc:2454] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 060ee373d5964855acdce98ab1b52197 in term 1.
I20260923 23:10:19.729907 21965 raft_consensus.cc:2781] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:19.730075 21965 raft_consensus.cc:686] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 1 LEADER]: Becoming Leader. State: Replica: 060ee373d5964855acdce98ab1b52197, State: Running, Role: LEADER
I20260923 23:10:19.730176 21965 consensus_queue.cc:237] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } }
I20260923 23:10:19.730996 21492 catalog_manager.cc:5495] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 reported cstate change: term changed from 0 to 1, leader changed from <none> to 060ee373d5964855acdce98ab1b52197 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "060ee373d5964855acdce98ab1b52197" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "060ee373d5964855acdce98ab1b52197" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38131 } health_report { overall_health: HEALTHY } } }
I20260923 23:10:19.735529 21955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.734723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52214 (local address 127.16.122.130:38131)
0923 23:10:19.734765 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.734766 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.734774 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:19.734831 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.734832 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.734842 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.734882 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.735390 (+ 508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.735391 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.735391 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.735394 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.735404 (+ 10us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.735407 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.735445 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.735475 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.735485 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:19.737000 21897 raft_consensus.cc:1261] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 1 FOLLOWER]: Refusing update from remote peer 060ee373d5964855acdce98ab1b52197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:19.737037 21635 raft_consensus.cc:1261] T 4e622a8220ad4e37b464abb41b9ed87c P 35b82a19a8bf44e18280b5e36074663c [term 1 FOLLOWER]: Refusing update from remote peer 060ee373d5964855acdce98ab1b52197: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:19.737304 21967 consensus_queue.cc:1035] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c92ecd98f107497db3daae5069db4cce" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 46301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:19.737422 21965 consensus_queue.cc:1035] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [LEADER]: Connected to new peer: Peer: permanent_uuid: "35b82a19a8bf44e18280b5e36074663c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 46541 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:19.739918 21971 mvcc.cc:205] Tried to move back new op lower bound from 7332679760840650752 to 7332679760815464448. Current Snapshot: MvccSnapshot[applied={T|T < 7332679760840650752}]
I20260923 23:10:19.742041 21491 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:19.742132 21491 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33202:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:19.742823 21491 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e282bb0bcf714ebaafc8f3ac6edc9453: Sending DeleteTablet for 3 replicas of tablet 4e622a8220ad4e37b464abb41b9ed87c
I20260923 23:10:19.743121 21615 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='slave'} at 127.0.0.1:49816
I20260923 23:10:19.743189 21746 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='slave'} at 127.0.0.1:53388
I20260923 23:10:19.743209 21877 tablet_service.cc:1514] Processing DeleteTablet for tablet 4e622a8220ad4e37b464abb41b9ed87c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:19 UTC) from {username='slave'} at 127.0.0.1:45028
I20260923 23:10:19.743444 21982 tablet_replica.cc:330] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: stopping tablet replica
I20260923 23:10:19.743543 21982 raft_consensus.cc:2229] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:19.743618 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21553
I20260923 23:10:19.743642 21982 raft_consensus.cc:2258] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:19.744089 21982 ts_tablet_manager.cc:1905] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:19.745592 21981 tablet_replica.cc:330] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: stopping tablet replica
I20260923 23:10:19.745639 21982 ts_tablet_manager.cc:1918] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:19.745692 21981 raft_consensus.cc:2229] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:19.745702 21982 log.cc:1196] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/4e622a8220ad4e37b464abb41b9ed87c
I20260923 23:10:19.745807 21981 raft_consensus.cc:2258] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:19.745939 21982 ts_tablet_manager.cc:1939] T 4e622a8220ad4e37b464abb41b9ed87c P c92ecd98f107497db3daae5069db4cce: Deleting consensus metadata
I20260923 23:10:19.746321 21478 catalog_manager.cc:4841] TS c92ecd98f107497db3daae5069db4cce (127.16.122.131:46301): tablet 4e622a8220ad4e37b464abb41b9ed87c (table test-table [id=57c5ebb9ba5644a0bd5a273eb491f1cf]) successfully deleted
W20260923 23:10:19.748390 21477 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:46541 (error 108)
I20260923 23:10:19.748487 21981 ts_tablet_manager.cc:1905] T 4e622a8220ad4e37b464abb41b9ed87c P 060ee373d5964855acdce98ab1b52197: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:19.749475 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21684
I20260923 23:10:19.750411 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.750297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:46541 (local address 127.0.0.1:58018)
0923 23:10:19.750334 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.750381 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:46541: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260923 23:10:19.750506 21477 catalog_manager.cc:4584] TS 35b82a19a8bf44e18280b5e36074663c (127.16.122.129:46541): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4e622a8220ad4e37b464abb41b9ed87c: Network error: Client connection negotiation failed: client connection to 127.16.122.129:46541: connect: Connection refused (error 111)
I20260923 23:10:19.754014 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21815
I20260923 23:10:19.754045 21951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.753945 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:38131 (local address 127.0.0.1:52220)
0923 23:10:19.753994 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.754019 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.130:38131: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:10:19.758227 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21462
2026-09-23T23:10:19Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (621 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260923 23:10:19.776716 16874 mini_oidc.cc:152] Starting JWKS server
I20260923 23:10:19.777751 16874 webserver.cc:460] Webserver started at https://127.0.0.1:41809/ using document root <none> and password file <none>
I20260923 23:10:19.777844 16874 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:10:19.778007 16874 webserver.cc:460] Webserver started at http://127.0.0.1:42763/ using document root <none> and password file <none>
2026-09-23T23:10:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:19Z Disabled control of system clock
I20260923 23:10:19.784905 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36777
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:34293
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:36777
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41809/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/testchainca.cert with env {}
W20260923 23:10:19.862349 21989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:19.862564 21989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:19.862584 21989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:19.863700 21989 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:19.863735 21989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:19.863746 21989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:19.863756 21989 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:19.863771 21989 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:10:19.863782 21989 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:10:19.863797 21989 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:41809/jwks.json
I20260923 23:10:19.865228 21989 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34293
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:36777
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36777
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:41809/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.21989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:19.865408 21989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:19.865577 21989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:19.870070 21994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.870087 21997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:19.870072 21995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:19.870136 21989 server_base.cc:1034] running on GCE node
I20260923 23:10:19.870472 21989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:19.870730 21989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:19.871862 21989 hybrid_clock.cc:648] HybridClock initialized: now 1790205019871844 us; error 34 us; skew 500 ppm
I20260923 23:10:19.873034 21989 webserver.cc:460] Webserver started at http://127.16.122.190:45815/ using document root <none> and password file <none>
I20260923 23:10:19.873240 21989 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:19.873284 21989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:19.873404 21989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:19.874423 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "798fb269821a4fe498fd900d0c956ef5"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "cfe516e61123aef2fc4d53253a077ff0"
server_key_iv: "2f7a3a6e3784bc5381630fdbed27c6da"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.874753 21989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "798fb269821a4fe498fd900d0c956ef5"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "cfe516e61123aef2fc4d53253a077ff0"
server_key_iv: "2f7a3a6e3784bc5381630fdbed27c6da"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.875958 21989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:19.876582 22003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:19.876715 21989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:19.876796 21989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "798fb269821a4fe498fd900d0c956ef5"
format_stamp: "Formatted at 2026-09-23 23:10:19 on dist-test-slave-95hh"
server_key: "cfe516e61123aef2fc4d53253a077ff0"
server_key_iv: "2f7a3a6e3784bc5381630fdbed27c6da"
server_key_version: "encryptionkey@0"
I20260923 23:10:19.876861 21989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:19.900400 21989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:19.900723 21989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:19.900852 21989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:19.904590 21989 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:36777
I20260923 23:10:19.904624 22055 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:36777 every 8 connection(s)
I20260923 23:10:19.904961 21989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:19.905879 22056 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:19.908218 22056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: Bootstrap starting.
I20260923 23:10:19.908633 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 21989
I20260923 23:10:19.908705 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:19.908811 22056 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:19.908864 16874 external_mini_cluster.cc:1428] Setting key e5cf3ccc3b0984d8d667790f102d55da
I20260923 23:10:19.909108 22056 log.cc:824] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:19.909837 22056 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: No bootstrap required, opened a new log
I20260923 23:10:19.911648 22056 raft_consensus.cc:348] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } }
I20260923 23:10:19.911785 22056 raft_consensus.cc:374] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:19.911816 22056 raft_consensus.cc:729] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 798fb269821a4fe498fd900d0c956ef5, State: Initialized, Role: FOLLOWER
I20260923 23:10:19.911949 22056 consensus_queue.cc:260] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } }
I20260923 23:10:19.912019 22056 raft_consensus.cc:388] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:19.912048 22056 raft_consensus.cc:482] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:19.912096 22056 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:19.912612 22056 raft_consensus.cc:504] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } }
I20260923 23:10:19.912705 22056 leader_election.cc:302] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 798fb269821a4fe498fd900d0c956ef5; no voters:
I20260923 23:10:19.912866 22056 leader_election.cc:288] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:19.912904 22061 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:19.913007 22061 raft_consensus.cc:686] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [term 1 LEADER]: Becoming Leader. State: Replica: 798fb269821a4fe498fd900d0c956ef5, State: Running, Role: LEADER
I20260923 23:10:19.913082 22061 consensus_queue.cc:237] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } }
I20260923 23:10:19.913203 22056 sys_catalog.cc:564] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:19.913398 22063 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "798fb269821a4fe498fd900d0c956ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } } }
I20260923 23:10:19.913563 22063 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:19.914181 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.909340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55298 (local address 127.16.122.190:36777)
0923 23:10:19.909552 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.909563 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.909600 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:10:19.910010 (+ 410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.910016 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.910087 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.910217 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:19.910409 (+ 192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.911317 (+ 908us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.912155 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.912558 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.912666 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.913352 (+ 686us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:19.913356 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:19.913356 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:19.913555 (+ 199us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:19.913599 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:19.913615 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:19.913675 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:19.913949 (+ 274us) server_negotiation.cc:299] Negotiation successful
0923 23:10:19.914043 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134,"thread_start_us":88,"threads_started":1}
I20260923 23:10:19.914804 22062 sys_catalog.cc:455] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 798fb269821a4fe498fd900d0c956ef5. Latest consensus state: current_term: 1 leader_uuid: "798fb269821a4fe498fd900d0c956ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "798fb269821a4fe498fd900d0c956ef5" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36777 } } }
I20260923 23:10:19.914875 22062 sys_catalog.cc:458] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5 [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:19.914913 22077 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:19.914959 22077 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:19.915004 22067 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:19.915136 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:10:19.915189 22067 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:19.916625 22067 catalog_manager.cc:1347] Generated new cluster ID: 1353e87b078d4032ae6b6421b1de25e2
I20260923 23:10:19.916666 22067 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:19.931272 22067 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:19.931993 22067 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:19.938218 22067 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 798fb269821a4fe498fd900d0c956ef5: Generated new TSK 0
I20260923 23:10:19.938434 22067 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 23:10:19.966189 22086 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260923 23:10:19.966353 22086 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 23:10:19.966630 22060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:19.944026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55312 (local address 127.16.122.190:36777)
0923 23:10:19.944074 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:19.944076 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:19.944169 (+ 93us) server_negotiation.cc:407] Connection header received
0923 23:10:19.944270 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:19.944271 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:19.944284 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:19.944339 (+ 55us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:10:19.944478 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.944982 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.945721 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.945934 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.945990 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.946185 (+ 195us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:10:19.966200 (+ 20015us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 23:10:19.966537 (+ 337us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:55312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":26}
W20260923 23:10:19.966737 22060 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:55312: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260923 23:10:19.966894 22085 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:19.943925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36777 (local address 127.0.0.1:55312)
0923 23:10:19.944103 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:19.944114 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:19.944184 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:19.944343 (+ 159us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:19.944345 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:19.944350 (+ 5us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:10:19.944414 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.944417 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.945063 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.945064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.945649 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:19.945651 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:19.945994 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:19.945995 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:19.946067 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:19.946088 (+ 21us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:10:19.966362 (+ 20274us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0923 23:10:19.966488 (+ 126us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.122.190:36777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":80,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.122.190:36777: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:41809/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260923 23:10:19.967329 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 21989
2026-09-23T23:10:19Z chronyd exiting
I20260923 23:10:20.002568 16874 test_util.cc:175] -----------------------------------------------
I20260923 23:10:20.002645 16874 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1790204999454807-16874-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (226 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260923 23:10:20.003417 16874 mini_oidc.cc:152] Starting JWKS server
I20260923 23:10:20.004340 16874 webserver.cc:460] Webserver started at https://127.0.0.1:35217/ using document root <none> and password file <none>
I20260923 23:10:20.004395 16874 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:10:20.004528 16874 webserver.cc:460] Webserver started at http://127.0.0.1:36707/ using document root <none> and password file <none>
2026-09-23T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:20Z Disabled control of system clock
I20260923 23:10:20.011408 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:44815
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:36923
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:44815
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35217/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/testchainca.cert with env {}
W20260923 23:10:20.088308 22092 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:20.088519 22092 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:20.088541 22092 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:20.089634 22092 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:20.089668 22092 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:20.089679 22092 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:20.089690 22092 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:20.089700 22092 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:10:20.089711 22092 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:10:20.089720 22092 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:35217/jwks.json
I20260923 23:10:20.091197 22092 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36923
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:44815
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:44815
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:35217/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22092
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:20.091406 22092 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:20.091568 22092 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:20.096082 22097 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:20.096088 22100 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:20.096122 22092 server_base.cc:1034] running on GCE node
W20260923 23:10:20.096091 22098 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:20.096501 22092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:20.096783 22092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:20.097947 22092 hybrid_clock.cc:648] HybridClock initialized: now 1790205020097924 us; error 37 us; skew 500 ppm
I20260923 23:10:20.099206 22092 webserver.cc:460] Webserver started at http://127.16.122.190:36995/ using document root <none> and password file <none>
I20260923 23:10:20.099426 22092 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:20.099476 22092 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:20.099596 22092 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:20.100617 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "ff26e4b3433d43d58150b7531307fc66"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "edf17a3bbd5c4d53d457bb690f473483"
server_key_iv: "7daf3d6226c9a9333ad1df75050bbd2b"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.100957 22092 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "ff26e4b3433d43d58150b7531307fc66"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "edf17a3bbd5c4d53d457bb690f473483"
server_key_iv: "7daf3d6226c9a9333ad1df75050bbd2b"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.102116 22092 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:20.102730 22106 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:20.102882 22092 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:20.102972 22092 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "ff26e4b3433d43d58150b7531307fc66"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "edf17a3bbd5c4d53d457bb690f473483"
server_key_iv: "7daf3d6226c9a9333ad1df75050bbd2b"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.103063 22092 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:20.113824 22092 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:20.114166 22092 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:20.114297 22092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:20.117916 22092 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:44815
I20260923 23:10:20.117955 22158 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:44815 every 8 connection(s)
I20260923 23:10:20.118283 22092 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:20.118919 22159 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:20.121618 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66: Bootstrap starting.
I20260923 23:10:20.122157 22159 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:20.122396 22159 log.cc:824] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:20.123056 22159 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66: No bootstrap required, opened a new log
I20260923 23:10:20.124617 22159 raft_consensus.cc:348] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } }
I20260923 23:10:20.124754 22159 raft_consensus.cc:374] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:20.124774 22159 raft_consensus.cc:729] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff26e4b3433d43d58150b7531307fc66, State: Initialized, Role: FOLLOWER
I20260923 23:10:20.124917 22159 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } }
I20260923 23:10:20.125000 22159 raft_consensus.cc:388] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:20.125036 22159 raft_consensus.cc:482] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:20.125083 22159 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:20.125082 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22092
I20260923 23:10:20.125168 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:20.125339 16874 external_mini_cluster.cc:1428] Setting key c7db501197766779fe7d9143256d1ea9
I20260923 23:10:20.125699 22159 raft_consensus.cc:504] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } }
I20260923 23:10:20.125802 22159 leader_election.cc:302] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ff26e4b3433d43d58150b7531307fc66; no voters:
I20260923 23:10:20.125942 22159 leader_election.cc:288] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:20.125981 22164 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:20.126118 22164 raft_consensus.cc:686] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [term 1 LEADER]: Becoming Leader. State: Replica: ff26e4b3433d43d58150b7531307fc66, State: Running, Role: LEADER
I20260923 23:10:20.126187 22159 sys_catalog.cc:564] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:20.126205 22164 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } }
I20260923 23:10:20.126621 22166 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff26e4b3433d43d58150b7531307fc66. Latest consensus state: current_term: 1 leader_uuid: "ff26e4b3433d43d58150b7531307fc66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } } }
I20260923 23:10:20.126706 22166 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:20.126847 22165 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ff26e4b3433d43d58150b7531307fc66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ff26e4b3433d43d58150b7531307fc66" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 44815 } } }
I20260923 23:10:20.126902 22165 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:20.126886 22172 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:20.127476 22172 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:20.129041 22172 catalog_manager.cc:1347] Generated new cluster ID: b1b611768da2463781f5a7b4e37bbaf8
I20260923 23:10:20.129091 22172 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:20.132040 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.125790 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39448 (local address 127.16.122.190:44815)
0923 23:10:20.126032 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.126080 (+ 48us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.126123 (+ 43us) server_negotiation.cc:407] Connection header received
0923 23:10:20.126372 (+ 249us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.126382 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.126477 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.127005 (+ 528us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:20.127148 (+ 143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.128536 (+ 1388us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.129589 (+ 1053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.130189 (+ 600us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.130296 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.131195 (+ 899us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:20.131200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:20.131200 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:20.131397 (+ 197us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:20.131447 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:20.131462 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:20.131510 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:20.131743 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 23:10:20.131821 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":80,"threads_started":1}
I20260923 23:10:20.132923 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:10:20.149235 22172 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:20.149968 22172 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:20.158231 22172 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ff26e4b3433d43d58150b7531307fc66: Generated new TSK 0
I20260923 23:10:20.158440 22172 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260923 23:10:20.181018 22189 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260923 23:10:20.181118 22189 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260923 23:10:20.181442 22163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.166980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39460 (local address 127.16.122.190:44815)
0923 23:10:20.167057 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.167059 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.167199 (+ 140us) server_negotiation.cc:407] Connection header received
0923 23:10:20.167280 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.167281 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.167292 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.167356 (+ 64us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:10:20.167493 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.167990 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.169110 (+ 1120us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.169325 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.169398 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.169578 (+ 180us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:10:20.181238 (+ 11660us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 23:10:20.181386 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:39460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":31}
W20260923 23:10:20.181490 22163 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:39460: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260923 23:10:20.181619 22188 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:20.166919 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:44815 (local address 127.0.0.1:39460)
0923 23:10:20.167128 (+ 209us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:20.167139 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:20.167211 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:20.167367 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:20.167368 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:20.167370 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:10:20.167436 (+ 66us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.167439 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.168056 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.168057 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.168988 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.168992 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.169412 (+ 420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.169413 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.169478 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.169484 (+ 6us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:10:20.181362 (+ 11878us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0923 23:10:20.181516 (+ 154us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.122.190:44815: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:35217/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":183,"thread_start_us":88,"threads_started":1}
I20260923 23:10:20.181958 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22092
2026-09-23T23:10:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (215 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260923 23:10:20.218914 16874 mini_oidc.cc:152] Starting JWKS server
I20260923 23:10:20.219656 16874 webserver.cc:460] Webserver started at https://127.0.0.1:43449/ using document root <none> and password file <none>
I20260923 23:10:20.219710 16874 mini_oidc.cc:189] Starting OIDC Discovery server
I20260923 23:10:20.219871 16874 webserver.cc:460] Webserver started at http://127.0.0.1:34085/ using document root <none> and password file <none>
2026-09-23T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:20Z Disabled control of system clock
I20260923 23:10:20.227005 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45709
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:34645
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:45709
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43449/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 23:10:20.306344 22195 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:20.306584 22195 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:20.306605 22195 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:20.307817 22195 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:20.307859 22195 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:20.307871 22195 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:20.307893 22195 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:20.307917 22195 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:10:20.307940 22195 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:10:20.307961 22195 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43449/jwks.json
I20260923 23:10:20.309340 22195 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34645
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:45709
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43449/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22195
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:20.309587 22195 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:20.309872 22195 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:20.314348 22203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:20.314378 22200 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:20.314400 22201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:20.314554 22195 server_base.cc:1034] running on GCE node
I20260923 23:10:20.314765 22195 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:20.315068 22195 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:20.316218 22195 hybrid_clock.cc:648] HybridClock initialized: now 1790205020316207 us; error 30 us; skew 500 ppm
I20260923 23:10:20.317395 22195 webserver.cc:460] Webserver started at http://127.16.122.190:42741/ using document root <none> and password file <none>
I20260923 23:10:20.317634 22195 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:20.317684 22195 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:20.317811 22195 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:20.318876 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "9bca8ab8e3734372b2a0873a90d75e09"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "4b93183f9a37d393f46637379d4c314e"
server_key_iv: "3faa33767a668802d928c7f22400ed0e"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.319242 22195 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "9bca8ab8e3734372b2a0873a90d75e09"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "4b93183f9a37d393f46637379d4c314e"
server_key_iv: "3faa33767a668802d928c7f22400ed0e"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.320514 22195 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:20.321177 22209 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:20.321333 22195 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:20.321414 22195 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "9bca8ab8e3734372b2a0873a90d75e09"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "4b93183f9a37d393f46637379d4c314e"
server_key_iv: "3faa33767a668802d928c7f22400ed0e"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.321496 22195 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:20.330729 22195 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:20.331019 22195 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:20.331192 22195 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:20.334728 22195 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:45709
I20260923 23:10:20.334760 22261 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:45709 every 8 connection(s)
I20260923 23:10:20.335119 22195 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:20.335726 22262 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:20.338423 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: Bootstrap starting.
I20260923 23:10:20.338992 22262 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:20.339284 22262 log.cc:824] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:20.339895 22262 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: No bootstrap required, opened a new log
I20260923 23:10:20.340487 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22195
I20260923 23:10:20.340583 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:20.340746 16874 external_mini_cluster.cc:1428] Setting key 61b93215b01df9b9de4c1d1db7661b64
I20260923 23:10:20.341674 22262 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } }
I20260923 23:10:20.341822 22262 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:20.341884 22262 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9bca8ab8e3734372b2a0873a90d75e09, State: Initialized, Role: FOLLOWER
I20260923 23:10:20.342010 22262 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } }
I20260923 23:10:20.342097 22262 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:20.342120 22262 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:20.342167 22262 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:20.342680 22262 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } }
I20260923 23:10:20.342763 22262 leader_election.cc:302] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9bca8ab8e3734372b2a0873a90d75e09; no voters:
I20260923 23:10:20.342902 22262 leader_election.cc:288] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:20.342944 22267 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:20.343115 22267 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [term 1 LEADER]: Becoming Leader. State: Replica: 9bca8ab8e3734372b2a0873a90d75e09, State: Running, Role: LEADER
I20260923 23:10:20.343180 22262 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:20.343207 22267 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } }
I20260923 23:10:20.343521 22267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9bca8ab8e3734372b2a0873a90d75e09. Latest consensus state: current_term: 1 leader_uuid: "9bca8ab8e3734372b2a0873a90d75e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } } }
I20260923 23:10:20.343580 22267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:20.343658 22268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9bca8ab8e3734372b2a0873a90d75e09" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9bca8ab8e3734372b2a0873a90d75e09" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45709 } } }
I20260923 23:10:20.343734 22268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09 [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:20.344993 22283 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:20.345077 22283 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:20.345156 22272 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:20.345356 22272 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:20.346705 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.341230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52674 (local address 127.16.122.190:45709)
0923 23:10:20.341423 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.341430 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.341457 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:10:20.341633 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.341639 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.341703 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.341812 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:20.342046 (+ 234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.343704 (+ 1658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.344669 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.345186 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.345599 (+ 413us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.346004 (+ 405us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:20.346008 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:20.346008 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:20.346169 (+ 161us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:20.346200 (+ 31us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:20.346211 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:20.346264 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:20.346449 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 23:10:20.346547 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126,"thread_start_us":76,"threads_started":1}
I20260923 23:10:20.346875 22272 catalog_manager.cc:1347] Generated new cluster ID: b63775b7294a4c20a4be58744fd455b7
I20260923 23:10:20.346935 22272 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:20.347616 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:10:20.358300 22272 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:20.359006 22272 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:20.366667 22272 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9bca8ab8e3734372b2a0873a90d75e09: Generated new TSK 0
I20260923 23:10:20.366851 22272 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:20.379297 22266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.375907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52682 (local address 127.16.122.190:45709)
0923 23:10:20.375950 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.375951 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.376033 (+ 82us) server_negotiation.cc:407] Connection header received
0923 23:10:20.376108 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.376109 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.376118 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.376175 (+ 57us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:10:20.376307 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.376788 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.377846 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.378022 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.378062 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.378224 (+ 162us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:10:20.379104 (+ 880us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43449/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 23:10:20.379227 (+ 123us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:52682: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43449/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:10:20.379406 22291 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:20.375820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45709 (local address 127.0.0.1:52682)
0923 23:10:20.375956 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:20.375967 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:20.376041 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:20.376189 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:20.376190 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:20.376192 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:10:20.376255 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.376257 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.376853 (+ 596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.376854 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.377770 (+ 916us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.377774 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.378083 (+ 309us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.378084 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.378141 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.378146 (+ 5us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:10:20.379207 (+ 1061us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43449/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0923 23:10:20.379303 (+ 96us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.122.190:45709: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43449/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":75,"threads_started":1}
I20260923 23:10:20.379624 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22195
E20260923 23:10:20.389859 22292 webserver.cc:533] Webserver: sslize failed:
E20260923 23:10:20.389942 22292 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
2026-09-23T23:10:20Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (197 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-09-23T23:10:20Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:20Z Disabled control of system clock
I20260923 23:10:20.423285 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41611
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:36475
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41611
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/testchainca.cert with env {}
W20260923 23:10:20.502216 22296 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:20.502439 22296 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:20.502470 22296 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:20.503691 22296 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:20.503734 22296 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:20.503751 22296 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:20.503767 22296 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:20.503782 22296 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260923 23:10:20.503805 22296 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260923 23:10:20.503822 22296 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260923 23:10:20.505306 22296 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36475
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41611
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41611
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22296
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:20.505563 22296 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:20.505756 22296 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:20.510241 22304 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:20.510358 22296 server_base.cc:1034] running on GCE node
W20260923 23:10:20.510257 22301 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:20.510251 22302 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:20.510687 22296 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:20.510972 22296 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:20.512112 22296 hybrid_clock.cc:648] HybridClock initialized: now 1790205020512098 us; error 39 us; skew 500 ppm
I20260923 23:10:20.513279 22296 webserver.cc:460] Webserver started at http://127.16.122.190:41559/ using document root <none> and password file <none>
I20260923 23:10:20.513495 22296 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:20.513545 22296 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:20.513662 22296 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:20.514672 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "c8618e408810414eb980b306599ed4cb"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "62d225a07bd682388c75740e24535f04"
server_key_iv: "b43df4b4ebfc95a27b99f8ce254029b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.514987 22296 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "c8618e408810414eb980b306599ed4cb"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "62d225a07bd682388c75740e24535f04"
server_key_iv: "b43df4b4ebfc95a27b99f8ce254029b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.516176 22296 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:20.516813 22310 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:20.516975 22296 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:20.517062 22296 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "c8618e408810414eb980b306599ed4cb"
format_stamp: "Formatted at 2026-09-23 23:10:20 on dist-test-slave-95hh"
server_key: "62d225a07bd682388c75740e24535f04"
server_key_iv: "b43df4b4ebfc95a27b99f8ce254029b3"
server_key_version: "encryptionkey@0"
I20260923 23:10:20.517125 22296 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:20.536803 22296 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:20.537129 22296 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:20.537257 22296 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:20.540776 22296 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41611
I20260923 23:10:20.540827 22362 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41611 every 8 connection(s)
I20260923 23:10:20.541146 22296 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:20.542050 22363 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:20.544296 22363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb: Bootstrap starting.
I20260923 23:10:20.544799 22363 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:20.545034 22363 log.cc:824] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:20.545696 22363 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb: No bootstrap required, opened a new log
I20260923 23:10:20.546926 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22296
I20260923 23:10:20.547019 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:20.547209 16874 external_mini_cluster.cc:1428] Setting key 48f80f8a51fca812a65f5e240e79752e
I20260923 23:10:20.547381 22363 raft_consensus.cc:348] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } }
I20260923 23:10:20.547536 22363 raft_consensus.cc:374] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:20.547570 22363 raft_consensus.cc:729] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c8618e408810414eb980b306599ed4cb, State: Initialized, Role: FOLLOWER
I20260923 23:10:20.547691 22363 consensus_queue.cc:260] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } }
I20260923 23:10:20.547770 22363 raft_consensus.cc:388] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:20.547813 22363 raft_consensus.cc:482] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:20.547868 22363 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:20.548350 22363 raft_consensus.cc:504] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } }
I20260923 23:10:20.548444 22363 leader_election.cc:302] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c8618e408810414eb980b306599ed4cb; no voters:
I20260923 23:10:20.548583 22363 leader_election.cc:288] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:20.548624 22368 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:20.548727 22368 raft_consensus.cc:686] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [term 1 LEADER]: Becoming Leader. State: Replica: c8618e408810414eb980b306599ed4cb, State: Running, Role: LEADER
I20260923 23:10:20.548823 22363 sys_catalog.cc:564] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:20.548812 22368 consensus_queue.cc:237] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } }
I20260923 23:10:20.549235 22369 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [sys.catalog]: SysCatalogTable state changed. Reason: New leader c8618e408810414eb980b306599ed4cb. Latest consensus state: current_term: 1 leader_uuid: "c8618e408810414eb980b306599ed4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } } }
I20260923 23:10:20.549363 22369 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:20.549528 22370 sys_catalog.cc:455] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c8618e408810414eb980b306599ed4cb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c8618e408810414eb980b306599ed4cb" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41611 } } }
I20260923 23:10:20.549623 22370 sys_catalog.cc:458] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:20.549763 22377 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:20.550246 22377 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:20.553030 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.547621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41796 (local address 127.16.122.190:41611)
0923 23:10:20.547823 (+ 202us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.547832 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.547866 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:10:20.547985 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.547991 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.548047 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.548160 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:20.548338 (+ 178us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.549997 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.550870 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.551640 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.551757 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.552313 (+ 556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:20.552316 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:20.552317 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:20.552541 (+ 224us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:20.552580 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:20.552594 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:20.552661 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:20.552744 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 23:10:20.552822 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260923 23:10:20.554215 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:10:20.559477 22377 catalog_manager.cc:1347] Generated new cluster ID: b953fb6d146c4c4b914e3eab8875e4d6
I20260923 23:10:20.559549 22377 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:20.585337 22377 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:20.585863 22377 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:20.589988 22377 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c8618e408810414eb980b306599ed4cb: Generated new TSK 0
I20260923 23:10:20.590168 22377 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:21.046628 22367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:20.600263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41806 (local address 127.16.122.190:41611)
0923 23:10:20.600309 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:20.600311 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:20.600384 (+ 73us) server_negotiation.cc:407] Connection header received
0923 23:10:20.600458 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:20.600458 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:20.600471 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:20.600574 (+ 103us) server_negotiation.cc:226] Negotiated authn=JWT
0923 23:10:20.600663 (+ 89us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.601427 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.602199 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.602497 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.602581 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.602708 (+ 127us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0923 23:10:21.046404 (+443696us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 23:10:21.046568 (+ 164us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 23:10:21.046700 22367 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41806: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260923 23:10:21.046775 22392 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:20.600170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41611 (local address 127.0.0.1:41806)
0923 23:10:20.600313 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:20.600324 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:20.600384 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:20.600572 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:20.600572 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:20.600574 (+ 2us) client_negotiation.cc:189] Negotiated authn=JWT
0923 23:10:20.600623 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.600624 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.601533 (+ 909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.601534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.602125 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:20.602127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:20.602577 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:20.602578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:20.602618 (+ 40us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:20.602622 (+ 4us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0923 23:10:21.046560 (+443938us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0923 23:10:21.046671 (+ 111us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.122.190:41611: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":85,"threads_started":1}
I20260923 23:10:21.047228 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22296
2026-09-23T23:10:21Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (647 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (85 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (84 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:21 dist-test-slave-95hh krb5kdc[22404](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:21 dist-test-slave-95hh krb5kdc[22404](info): set up 2 sockets
Sep 23 23:10:21 dist-test-slave-95hh krb5kdc[22404](info): commencing operation
krb5kdc: starting...
W20260923 23:10:22.167649 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:22.279560 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43243
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:43243
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:22.359261 22417 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:22.359485 22417 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:22.359508 22417 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:22.360750 22417 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:22.360806 22417 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:22.360819 22417 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:22.360831 22417 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:22.360842 22417 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:22.362380 22417 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:43243
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22417
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:22.362632 22417 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:22.362855 22417 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:22.367601 22422 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:22.367683 22425 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.367640 22417 server_base.cc:1034] running on GCE node
W20260923 23:10:22.367611 22423 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.368108 22417 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:22.368391 22417 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:22.369531 22417 hybrid_clock.cc:648] HybridClock initialized: now 1790205022369521 us; error 27 us; skew 500 ppm
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.374696 22417 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:22.375180 22417 webserver.cc:460] Webserver started at http://127.16.122.190:36547/ using document root <none> and password file <none>
I20260923 23:10:22.375408 22417 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:22.375464 22417 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:22.375592 22417 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:22.376627 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "5392bac5e6b740c6ab6b12e8210d2aa6"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "94ba70bc0f0f101bca29a8e83fa42333"
server_key_iv: "d45c1ea7af8f387eb6daa911b2a3e299"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.376961 22417 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "5392bac5e6b740c6ab6b12e8210d2aa6"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "94ba70bc0f0f101bca29a8e83fa42333"
server_key_iv: "d45c1ea7af8f387eb6daa911b2a3e299"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.378153 22417 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:22.378798 22432 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.378937 22417 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:22.379014 22417 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "5392bac5e6b740c6ab6b12e8210d2aa6"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "94ba70bc0f0f101bca29a8e83fa42333"
server_key_iv: "d45c1ea7af8f387eb6daa911b2a3e299"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.379103 22417 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:22.403316 22417 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:22.405443 22417 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:22.405591 22417 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:22.409430 22417 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:43243
I20260923 23:10:22.409523 22484 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:43243 every 8 connection(s)
I20260923 23:10:22.409826 22417 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:22.410686 22485 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:22.413003 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6: Bootstrap starting.
I20260923 23:10:22.413517 22485 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:22.413749 22485 log.cc:824] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:22.414373 22485 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6: No bootstrap required, opened a new log
I20260923 23:10:22.414494 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22417
I20260923 23:10:22.414594 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:22.414783 16874 external_mini_cluster.cc:1428] Setting key be905a9625253a31e00382c2158e0919
I20260923 23:10:22.416337 22485 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } }
I20260923 23:10:22.416481 22485 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:22.416512 22485 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5392bac5e6b740c6ab6b12e8210d2aa6, State: Initialized, Role: FOLLOWER
I20260923 23:10:22.416673 22485 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } }
I20260923 23:10:22.416759 22485 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:22.416795 22485 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:22.416841 22485 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:22.417790 22485 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } }
I20260923 23:10:22.417881 22485 leader_election.cc:302] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5392bac5e6b740c6ab6b12e8210d2aa6; no voters:
I20260923 23:10:22.418032 22485 leader_election.cc:288] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:22.418068 22490 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:22.418207 22490 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [term 1 LEADER]: Becoming Leader. State: Replica: 5392bac5e6b740c6ab6b12e8210d2aa6, State: Running, Role: LEADER
I20260923 23:10:22.418286 22485 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:22.418308 22490 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } }
I20260923 23:10:22.418642 22490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5392bac5e6b740c6ab6b12e8210d2aa6. Latest consensus state: current_term: 1 leader_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } } }
I20260923 23:10:22.418850 22490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:22.418787 22491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5392bac5e6b740c6ab6b12e8210d2aa6" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 43243 } } }
I20260923 23:10:22.419274 22491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:22.419411 22499 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:22.419950 22499 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:22.421495 22499 catalog_manager.cc:1347] Generated new cluster ID: d18d6aa86bbc4e78bc46ae484d5480e1
I20260923 23:10:22.421545 22499 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:22.426180 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.415315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39756 (local address 127.16.122.190:43243)
0923 23:10:22.415496 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.415505 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.415530 (+ 25us) server_negotiation.cc:407] Connection header received
0923 23:10:22.416320 (+ 790us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.416329 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.416385 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.416540 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.417281 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.418359 (+ 1078us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.419375 (+ 1016us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.419938 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.420029 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.422649 (+ 2620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.422653 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.422653 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.422665 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.424590 (+ 1925us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.424942 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.424945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.424945 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.425067 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.425236 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.425237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.425237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.425559 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.425734 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.425812 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.425967 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127,"thread_start_us":74,"threads_started":1}
I20260923 23:10:22.432519 22499 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:22.433030 22499 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:22.453783 22499 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5392bac5e6b740c6ab6b12e8210d2aa6: Generated new TSK 0
I20260923 23:10:22.453977 22499 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:22.485406 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:22.565044 22511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:22.565254 22511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:22.565277 22511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:22.566452 22511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:22.566494 22511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:22.566520 22511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:22.568058 22511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22511
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:22.568284 22511 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:22.568511 22511 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:22.573367 22516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:22.573385 22517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.573397 22511 server_base.cc:1034] running on GCE node
W20260923 23:10:22.573379 22519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.573736 22511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:22.573961 22511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:22.575093 22511 hybrid_clock.cc:648] HybridClock initialized: now 1790205022575077 us; error 29 us; skew 500 ppm
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.577596 22511 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:22.578047 22511 webserver.cc:460] Webserver started at http://127.16.122.129:33651/ using document root <none> and password file <none>
I20260923 23:10:22.578244 22511 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:22.578290 22511 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:22.578411 22511 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:22.579501 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "7064422625aa497b8f35f7d1fb4a23bf"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "866cb579d9ee965ae1a21b6c0e2b7978"
server_key_iv: "a21aae0c7bbd22399af4a5e5cdf7af75"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.579828 22511 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "7064422625aa497b8f35f7d1fb4a23bf"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "866cb579d9ee965ae1a21b6c0e2b7978"
server_key_iv: "a21aae0c7bbd22399af4a5e5cdf7af75"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.581090 22511 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:22.581808 22526 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.581955 22511 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:22.582037 22511 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "7064422625aa497b8f35f7d1fb4a23bf"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "866cb579d9ee965ae1a21b6c0e2b7978"
server_key_iv: "a21aae0c7bbd22399af4a5e5cdf7af75"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.582098 22511 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:22.600075 22511 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:22.602015 22511 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:22.602159 22511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:22.602377 22511 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:22.602679 22511 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:22.602713 22511 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.602741 22511 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:22.602758 22511 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.608225 22511 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:46255
I20260923 23:10:22.608266 22639 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:46255 every 8 connection(s)
I20260923 23:10:22.608584 22511 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:22.610069 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22511
I20260923 23:10:22.610167 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:22.610328 16874 external_mini_cluster.cc:1428] Setting key ac469f53f3c4bc70cb88314624015352
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:10:22.618613 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.609343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:35371 (local address 127.16.122.190:43243)
0923 23:10:22.609412 (+ 69us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.609414 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.609590 (+ 176us) server_negotiation.cc:407] Connection header received
0923 23:10:22.610735 (+ 1145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.610736 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.610752 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.610797 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.611545 (+ 748us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.612086 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.613095 (+ 1009us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.613383 (+ 288us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.613452 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.615695 (+ 2243us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.615697 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.615697 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.615705 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.617288 (+ 1583us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.617693 (+ 405us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.617693 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.617694 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.617755 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.618022 (+ 267us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.618022 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.618023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.618160 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.618223 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.618470 (+ 247us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.618543 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20260923 23:10:22.618768 22642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.609307 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43243 (local address 127.16.122.129:35371)
0923 23:10:22.609471 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:22.609513 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:22.610638 (+ 1125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:22.610884 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:22.610889 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:22.611247 (+ 358us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:22.611442 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.611447 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.612184 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.612185 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.613006 (+ 821us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.613008 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.613457 (+ 449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.613457 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.613555 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.614127 (+ 572us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:22.614136 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:22.615595 (+ 1459us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:22.617384 (+ 1789us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.617391 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.617393 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.617621 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.617819 (+ 198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.617820 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.617820 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.617956 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.618219 (+ 263us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:22.618224 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:22.618351 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 23:10:22.618443 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 23:10:22.618570 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":84,"threads_started":1}
I20260923 23:10:22.619333 22640 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43243
I20260923 23:10:22.619429 22640 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:22.619611 22640 heartbeater.cc:511] Master 127.16.122.190:43243 requested a full tablet report, sending...
I20260923 23:10:22.620208 22449 ts_manager.cc:194] Registered new tserver with Master: 7064422625aa497b8f35f7d1fb4a23bf (127.16.122.129:46255)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
I20260923 23:10:22.620945 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:35371
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:22.634241 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:22.712878 22645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:22.713089 22645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:22.713114 22645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:22.714226 22645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:22.714258 22645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:22.714283 22645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:22.715735 22645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:22.715937 22645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:22.716115 22645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:22.720921 22651 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:22.720888 22650 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.721107 22645 server_base.cc:1034] running on GCE node
W20260923 23:10:22.720916 22653 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.721406 22645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:22.721637 22645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:22.722775 22645 hybrid_clock.cc:648] HybridClock initialized: now 1790205022722764 us; error 30 us; skew 500 ppm
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.725135 22645 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:22.725565 22645 webserver.cc:460] Webserver started at http://127.16.122.130:35881/ using document root <none> and password file <none>
I20260923 23:10:22.725755 22645 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:22.725801 22645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:22.725919 22645 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:22.726897 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "45b5c3fc0b0a4d11be501a1e6afced8e"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "fc113103e1b43991578d8ccd9557d448"
server_key_iv: "06af03991ab0014de70cbecf92748b82"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.727268 22645 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "45b5c3fc0b0a4d11be501a1e6afced8e"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "fc113103e1b43991578d8ccd9557d448"
server_key_iv: "06af03991ab0014de70cbecf92748b82"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.728497 22645 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:22.729127 22660 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.729285 22645 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:22.729364 22645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "45b5c3fc0b0a4d11be501a1e6afced8e"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "fc113103e1b43991578d8ccd9557d448"
server_key_iv: "06af03991ab0014de70cbecf92748b82"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.729422 22645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:22.755654 22645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:22.757395 22645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:22.757541 22645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:22.757761 22645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:22.758059 22645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:22.758090 22645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.758118 22645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:22.758141 22645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.763414 22645 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:37615
I20260923 23:10:22.763489 22773 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:37615 every 8 connection(s)
I20260923 23:10:22.763784 22645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:22.769167 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22645
I20260923 23:10:22.769258 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:22.769439 16874 external_mini_cluster.cc:1428] Setting key d63b1b29cb9e13bb7da7a6e7bf7dfe62
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:22.772972 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.764558 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:46143 (local address 127.16.122.190:43243)
0923 23:10:22.764622 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.764624 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.764832 (+ 208us) server_negotiation.cc:407] Connection header received
0923 23:10:22.765744 (+ 912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.765745 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.765757 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.765808 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.766415 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.766913 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.767931 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.768135 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.768190 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.770365 (+ 2175us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.770367 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.770367 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.770375 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.771730 (+ 1355us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.772097 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.772098 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.772099 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.772159 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.772349 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.772349 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.772350 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.772472 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.772535 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.772789 (+ 254us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.772873 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:10:22.773128 22776 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.764547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43243 (local address 127.16.122.130:46143)
0923 23:10:22.764712 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:22.764748 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:22.765638 (+ 890us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:22.765882 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:22.765885 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:22.766135 (+ 250us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:22.766323 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.766327 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.766998 (+ 671us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.766999 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.767844 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.767846 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.768206 (+ 360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.768207 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.768322 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.768791 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:22.768799 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:22.770282 (+ 1483us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:22.771809 (+ 1527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.771815 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.771817 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.772033 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.772211 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.772212 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.772212 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.772299 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.772529 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:22.772536 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:22.772673 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 23:10:22.772761 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:10:22.772910 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260923 23:10:22.773495 22774 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43243
I20260923 23:10:22.773589 22774 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:22.773756 22774 heartbeater.cc:511] Master 127.16.122.190:43243 requested a full tablet report, sending...
I20260923 23:10:22.774104 22449 ts_manager.cc:194] Registered new tserver with Master: 45b5c3fc0b0a4d11be501a1e6afced8e (127.16.122.130:37615)
I20260923 23:10:22.774462 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:46143
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:22.794025 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:22.872509 22779 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:22.872722 22779 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:22.872753 22779 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:22.873905 22779 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:22.873947 22779 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:22.873989 22779 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:22.875494 22779 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:43243
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22779
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:22.875728 22779 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:22.875942 22779 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:22.880577 22784 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.880694 22779 server_base.cc:1034] running on GCE node
W20260923 23:10:22.880584 22787 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:22.880788 22785 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:22.881014 22779 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:22.881248 22779 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:22.882377 22779 hybrid_clock.cc:648] HybridClock initialized: now 1790205022882366 us; error 30 us; skew 500 ppm
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.884711 22779 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:22.885155 22779 webserver.cc:460] Webserver started at http://127.16.122.131:43013/ using document root <none> and password file <none>
I20260923 23:10:22.885324 22779 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:22.885365 22779 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:22.885449 22779 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:22.886425 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "30839e170339431da1130314fd099658"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "1e59052d69ec311041c7c29fd5629e13"
server_key_iv: "18c2bd71db2cbb6b1e2316c808fbd430"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.886703 22779 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "30839e170339431da1130314fd099658"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "1e59052d69ec311041c7c29fd5629e13"
server_key_iv: "18c2bd71db2cbb6b1e2316c808fbd430"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.887976 22779 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:22.888615 22794 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.888746 22779 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:22.888818 22779 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "30839e170339431da1130314fd099658"
format_stamp: "Formatted at 2026-09-23 23:10:22 on dist-test-slave-95hh"
server_key: "1e59052d69ec311041c7c29fd5629e13"
server_key_iv: "18c2bd71db2cbb6b1e2316c808fbd430"
server_key_version: "encryptionkey@0"
I20260923 23:10:22.888880 22779 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:22.904956 22779 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:22.906864 22779 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:22.906991 22779 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:22.907256 22779 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:22.907563 22779 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:22.907598 22779 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.907627 22779 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:22.907649 22779 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:22.912995 22779 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:37959
I20260923 23:10:22.913055 22907 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:37959 every 8 connection(s)
I20260923 23:10:22.913334 22779 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:22.918558 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22779
I20260923 23:10:22.918668 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:22.918875 16874 external_mini_cluster.cc:1428] Setting key 34732f0743c61b3a6bede8b5ff48b439
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:22.922058 22489 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.913993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:57345 (local address 127.16.122.190:43243)
0923 23:10:22.914050 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.914052 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.914253 (+ 201us) server_negotiation.cc:407] Connection header received
0923 23:10:22.915140 (+ 887us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.915141 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.915151 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.915201 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.915764 (+ 563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.916248 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.917239 (+ 991us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.917424 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.917482 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.919890 (+ 2408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.919892 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.919892 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.919898 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.920993 (+ 1095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.921342 (+ 349us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.921343 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.921343 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.921393 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.921583 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.921584 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.921584 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.921684 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.921730 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.921936 (+ 206us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.921994 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:22.922215 22910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.913969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:43243 (local address 127.16.122.131:57345)
0923 23:10:22.914125 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:22.914187 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:22.915064 (+ 877us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:22.915264 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:22.915267 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:22.915519 (+ 252us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:22.915704 (+ 185us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.915708 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.916323 (+ 615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.916323 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.917154 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:22.917157 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.917510 (+ 353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.917511 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:22.917672 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.918107 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:22.918115 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:22.919819 (+ 1704us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:22.921059 (+ 1240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.921064 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.921066 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.921285 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.921444 (+ 159us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:22.921445 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:22.921445 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:22.921541 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:22.921726 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:22.921728 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:22.921845 (+ 117us) client_negotiation.cc:769] Sending connection context
0923 23:10:22.921921 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:10:22.922049 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":79,"threads_started":1}
I20260923 23:10:22.922623 22908 heartbeater.cc:347] Connected to a master server at 127.16.122.190:43243
I20260923 23:10:22.922710 22908 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:22.922886 22908 heartbeater.cc:511] Master 127.16.122.190:43243 requested a full tablet report, sending...
I20260923 23:10:22.923256 22449 ts_manager.cc:194] Registered new tserver with Master: 30839e170339431da1130314fd099658 (127.16.122.131:37959)
I20260923 23:10:22.923635 22449 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:57345
I20260923 23:10:22.930697 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/test-admin.keytab.
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.945366 16874 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260923 23:10:22.946867 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:22.946172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48614 (local address 127.16.122.129:46255)
0923 23:10:22.946347 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.946352 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.946363 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:22.946389 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.946391 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.946404 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.946486 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.946802 (+ 316us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48614: BlockingRecv error: recv got EOF from 127.0.0.1:48614 (error 108)
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":85,"threads_started":1}
W20260923 23:10:22.946992 16874 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc' of type FILE
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.947983 16874 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.948788 16874 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:22.956139 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.949252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48628 (local address 127.16.122.129:46255)
0923 23:10:22.949292 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.949295 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.949323 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:10:22.949390 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.949391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.949402 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.949447 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.949814 (+ 367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.950532 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.951260 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.951597 (+ 337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.951655 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.953726 (+ 2071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.953738 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.953738 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.953746 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.955214 (+ 1468us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.955529 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.955532 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.955532 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.955582 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.955719 (+ 137us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.955719 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.955719 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.955839 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.955936 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.956003 (+ 67us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.956062 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:10:22.957734 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:22.957248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48638 (local address 127.16.122.129:46255)
0923 23:10:22.957285 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.957286 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.957334 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:10:22.957389 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.957389 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.957399 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.957469 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.957691 (+ 222us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48638: BlockingRecv error: recv got EOF from 127.0.0.1:48638 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:10:22.958011 16874 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc)
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.958892 16874 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.959640 16874 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:22.965634 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.960073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48654 (local address 127.16.122.129:46255)
0923 23:10:22.960110 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.960112 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.960137 (+ 25us) server_negotiation.cc:407] Connection header received
0923 23:10:22.960186 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.960187 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.960195 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.960235 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.960533 (+ 298us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.961001 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.961635 (+ 634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.961842 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.961906 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.963707 (+ 1801us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.963708 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.963709 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.963714 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.964751 (+ 1037us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.965009 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.965010 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.965010 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.965052 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.965263 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.965264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.965264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.965338 (+ 74us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.965420 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.965524 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.965576 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:10:22.966920 22915 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:22.966475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48668 (local address 127.16.122.129:46255)
0923 23:10:22.966513 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.966515 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.966544 (+ 29us) server_negotiation.cc:407] Connection header received
0923 23:10:22.966606 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.966606 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.966615 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.966656 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.966877 (+ 221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48668: BlockingRecv error: recv got EOF from 127.0.0.1:48668 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:10:22.966989 16874 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1790204999454807-16874-0/krb5kdc/krb5cc' of type FILE
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.967881 16874 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:22.968595 16874 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 23 23:10:22 dist-test-slave-95hh krb5kdc[22404](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205022, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:10:22.974647 22915 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:22.968993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48678 (local address 127.16.122.129:46255)
0923 23:10:22.969034 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:22.969036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:22.969087 (+ 51us) server_negotiation.cc:407] Connection header received
0923 23:10:22.969121 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:22.969122 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:22.969128 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:22.969174 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:22.969458 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.969893 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.970604 (+ 711us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:22.970780 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:22.970834 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:22.972711 (+ 1877us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:22.972712 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:22.972712 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:22.972718 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:22.973869 (+ 1151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.974145 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.974145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.974146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.974187 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:22.974303 (+ 116us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:22.974304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:22.974304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:22.974376 (+ 72us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:22.974445 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:22.974511 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 23:10:22.974563 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:10:22.975253 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22511
I20260923 23:10:22.980005 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22645
I20260923 23:10:22.983731 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22779
I20260923 23:10:22.987460 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22417
2026-09-23T23:10:22Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (1770 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:23 dist-test-slave-95hh krb5kdc[22928](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:23 dist-test-slave-95hh krb5kdc[22928](info): set up 2 sockets
Sep 23 23:10:23 dist-test-slave-95hh krb5kdc[22928](info): commencing operation
krb5kdc: starting...
W20260923 23:10:23.943764 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:24Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab.
Entry for principal oryx/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab.
I20260923 23:10:24.055212 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:39045
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab
--principal=oryx/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39045
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:24.133487 22941 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:24.133709 22941 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:24.133740 22941 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:24.134851 22941 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:24.134891 22941 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:24.134908 22941 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:24.134923 22941 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:24.134938 22941 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:24.136374 22941 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39045
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab
--principal=oryx/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:39045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.22941
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:24.136613 22941 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:24.136811 22941 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:24.141225 22946 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.141232 22947 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.141232 22949 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.141340 22941 server_base.cc:1034] running on GCE node
I20260923 23:10:24.141743 22941 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.141983 22941 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.143082 22941 hybrid_clock.cc:648] HybridClock initialized: now 1790205024143066 us; error 27 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.145340 22941 init.cc:377] Logged in from keytab as oryx/127.16.122.190@KRBTEST.COM (short username oryx)
I20260923 23:10:24.145776 22941 webserver.cc:460] Webserver started at http://127.16.122.190:36099/ using document root <none> and password file <none>
I20260923 23:10:24.145975 22941 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.146021 22941 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.146139 22941 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:24.147205 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "a3802b5cce894104b40cb8e29fc478fc"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "b254458461cc6f7b95e2575eff926666"
server_key_iv: "4a9c8345d06edff50d6a57b25b6ac352"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.147518 22941 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "a3802b5cce894104b40cb8e29fc478fc"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "b254458461cc6f7b95e2575eff926666"
server_key_iv: "4a9c8345d06edff50d6a57b25b6ac352"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.148694 22941 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:24.149292 22956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.149435 22941 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.149511 22941 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "a3802b5cce894104b40cb8e29fc478fc"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "b254458461cc6f7b95e2575eff926666"
server_key_iv: "4a9c8345d06edff50d6a57b25b6ac352"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.149566 22941 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.164258 22941 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.165958 22941 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:24.166092 22941 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.169865 22941 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:39045
I20260923 23:10:24.169898 23008 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:39045 every 8 connection(s)
I20260923 23:10:24.170222 22941 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:24.171187 23009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:24.173339 23009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Bootstrap starting.
I20260923 23:10:24.173866 23009 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:24.174124 23009 log.cc:824] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:24.174733 23009 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: No bootstrap required, opened a new log
I20260923 23:10:24.176251 23009 raft_consensus.cc:348] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } }
I20260923 23:10:24.176383 23009 raft_consensus.cc:374] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:24.176404 23009 raft_consensus.cc:729] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a3802b5cce894104b40cb8e29fc478fc, State: Initialized, Role: FOLLOWER
I20260923 23:10:24.176530 23009 consensus_queue.cc:260] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } }
I20260923 23:10:24.176597 23009 raft_consensus.cc:388] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:24.176620 23009 raft_consensus.cc:482] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:24.176666 23009 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:24.177384 23009 raft_consensus.cc:504] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } }
I20260923 23:10:24.177471 23009 leader_election.cc:302] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a3802b5cce894104b40cb8e29fc478fc; no voters:
I20260923 23:10:24.177616 23009 leader_election.cc:288] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:24.177654 23012 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:24.177784 23012 raft_consensus.cc:686] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 LEADER]: Becoming Leader. State: Replica: a3802b5cce894104b40cb8e29fc478fc, State: Running, Role: LEADER
I20260923 23:10:24.177842 23009 sys_catalog.cc:564] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:24.177867 23012 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } }
I20260923 23:10:24.178181 23013 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } }
I20260923 23:10:24.178198 23014 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3802b5cce894104b40cb8e29fc478fc. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } }
I20260923 23:10:24.178336 23013 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:24.178347 23014 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:24.178776 23021 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:24.179286 23021 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:24.179804 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 22941
I20260923 23:10:24.179898 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:24.180045 16874 external_mini_cluster.cc:1428] Setting key 987e6fae4be64551bfc87d74d5b84c4c
I20260923 23:10:24.180770 23021 catalog_manager.cc:1347] Generated new cluster ID: e497fe61c14b41858bb5d98af5ca70d5
I20260923 23:10:24.180809 23021 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:24.185886 23021 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:24.186554 23021 catalog_manager.cc:1504] Loading token signing keys...
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:24.195845 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.180574 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53570 (local address 127.16.122.190:39045)
0923 23:10:24.185692 (+ 5118us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.185701 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.185738 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:10:24.185856 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.185863 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.185919 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.186084 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.186796 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.188228 (+ 1432us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.189191 (+ 963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.189806 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.190010 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.192489 (+ 2479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.192493 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.192493 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.192505 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.194275 (+ 1770us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.194643 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.194646 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.194646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.194740 (+ 94us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.194935 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.194936 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.194936 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.195244 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.195399 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.195478 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.195627 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5075,"thread_start_us":65,"threads_started":1}
I20260923 23:10:24.197119 23021 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Generated new TSK 0
I20260923 23:10:24.197269 23021 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
I20260923 23:10:24.221812 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:24.302258 23035 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:24.302470 23035 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:24.302495 23035 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:24.303678 23035 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:24.303718 23035 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:24.303743 23035 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:24.305146 23035 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23035
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:24.305334 23035 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:24.305523 23035 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:24.310272 23043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.310350 23041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.310387 23040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.310280 23035 server_base.cc:1034] running on GCE node
I20260923 23:10:24.310653 23035 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.310881 23035 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.312041 23035 hybrid_clock.cc:648] HybridClock initialized: now 1790205024312024 us; error 33 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.314446 23035 init.cc:377] Logged in from keytab as oryx/127.16.122.129@KRBTEST.COM (short username oryx)
I20260923 23:10:24.314882 23035 webserver.cc:460] Webserver started at http://127.16.122.129:39773/ using document root <none> and password file <none>
I20260923 23:10:24.315127 23035 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.315178 23035 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.315310 23035 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:24.316361 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "6912c866b06549f2b43547471502e863"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "38ac62004b4d06f5f5d62dc1da721e5b"
server_key_iv: "7538b507ea510076c42afebda586222c"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.316704 23035 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "6912c866b06549f2b43547471502e863"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "38ac62004b4d06f5f5d62dc1da721e5b"
server_key_iv: "7538b507ea510076c42afebda586222c"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.317974 23035 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:24.318655 23050 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.318804 23035 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.318886 23035 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "6912c866b06549f2b43547471502e863"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "38ac62004b4d06f5f5d62dc1da721e5b"
server_key_iv: "7538b507ea510076c42afebda586222c"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.318956 23035 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.331770 23035 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.333905 23035 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:24.334038 23035 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.334255 23035 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:24.334563 23035 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:24.334596 23035 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.334625 23035 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:24.334647 23035 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.340231 23035 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:41871
I20260923 23:10:24.340269 23163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:41871 every 8 connection(s)
I20260923 23:10:24.340592 23035 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:24.346544 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23035
I20260923 23:10:24.346649 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:24.346855 16874 external_mini_cluster.cc:1428] Setting key 1286482a61672cdfdffc07ebf0583471
I20260923 23:10:24.349961 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.341348 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50851 (local address 127.16.122.190:39045)
0923 23:10:24.341390 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.341392 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.341598 (+ 206us) server_negotiation.cc:407] Connection header received
0923 23:10:24.342133 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.342134 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.342146 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.342200 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.343188 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.343696 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.344677 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.344874 (+ 197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.344928 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.347231 (+ 2303us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.347232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.347233 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.347241 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.348499 (+ 1258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.348978 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.348979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.348980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.349043 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.349299 (+ 256us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.349300 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.349300 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.349402 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.349450 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.349776 (+ 326us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.349870 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:10:24.350136 23166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.341311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:50851)
0923 23:10:24.341476 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.341513 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.342034 (+ 521us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.342281 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.342285 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.342859 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.343104 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.343111 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.343778 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.343780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.344593 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.344595 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.344942 (+ 347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.344943 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.345078 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.345559 (+ 481us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.345568 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.347156 (+ 1588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.348584 (+ 1428us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.348592 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.348594 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.348896 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.349106 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.349106 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.349107 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.349244 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.349455 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.349460 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.349632 (+ 172us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.349747 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.349907 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":88,"threads_started":1}
I20260923 23:10:24.350584 23164 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:24.350673 23164 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:24.350843 23164 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:24.351356 22973 ts_manager.cc:194] Registered new tserver with Master: 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:24.351970 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:50851
WARNING: no policy specified for oryx/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
I20260923 23:10:24.369619 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:24.448910 23169 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:24.449111 23169 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:24.449141 23169 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:24.450261 23169 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:24.450299 23169 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:24.450323 23169 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:24.451779 23169 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23169
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:24.451964 23169 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:24.452144 23169 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:24.456791 23177 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.456799 23174 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.456981 23169 server_base.cc:1034] running on GCE node
W20260923 23:10:24.456799 23175 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.457255 23169 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.457470 23169 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.458598 23169 hybrid_clock.cc:648] HybridClock initialized: now 1790205024458587 us; error 29 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.460974 23169 init.cc:377] Logged in from keytab as oryx/127.16.122.130@KRBTEST.COM (short username oryx)
I20260923 23:10:24.461400 23169 webserver.cc:460] Webserver started at http://127.16.122.130:46429/ using document root <none> and password file <none>
I20260923 23:10:24.461592 23169 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.461637 23169 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.461730 23169 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:24.462716 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "aaa84a1d27d04e4596a3768e2953b51b"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "29a150b052d445f77c98e67aecee4e17"
server_key_iv: "e91688b7cc4054436cf626cdc4ccc833"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.463006 23169 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "aaa84a1d27d04e4596a3768e2953b51b"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "29a150b052d445f77c98e67aecee4e17"
server_key_iv: "e91688b7cc4054436cf626cdc4ccc833"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.464255 23169 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:24.464892 23184 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.465041 23169 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:24.465114 23169 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "aaa84a1d27d04e4596a3768e2953b51b"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "29a150b052d445f77c98e67aecee4e17"
server_key_iv: "e91688b7cc4054436cf626cdc4ccc833"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.465181 23169 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.476022 23169 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.478127 23169 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:24.478257 23169 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.478492 23169 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:24.478804 23169 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:24.478837 23169 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.478860 23169 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:24.478883 23169 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.484290 23169 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:44843
I20260923 23:10:24.484349 23297 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:44843 every 8 connection(s)
I20260923 23:10:24.484658 23169 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:24.493175 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.485369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:42245 (local address 127.16.122.190:39045)
0923 23:10:24.485430 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.485432 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.485622 (+ 190us) server_negotiation.cc:407] Connection header received
0923 23:10:24.486076 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.486078 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.486089 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.486151 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.487026 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.487547 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.488489 (+ 942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.488671 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.488723 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.490808 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.490810 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.490810 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.490817 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.492040 (+ 1223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.492379 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.492380 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.492380 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.492457 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.492658 (+ 201us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.492659 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.492659 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.492770 (+ 111us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.492825 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.493036 (+ 211us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.493104 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:24.493325 23300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.485328 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:42245)
0923 23:10:24.485472 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.485514 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.485991 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.486204 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.486208 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.486757 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.486952 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.486957 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.487623 (+ 666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.487624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.488412 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.488415 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.488734 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.488734 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.488830 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.489333 (+ 503us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.489341 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.490719 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.492107 (+ 1388us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.492114 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.492116 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.492322 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.492505 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.492505 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.492506 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.492596 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.492818 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.492821 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.492935 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.493013 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.493135 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260923 23:10:24.493767 23298 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:24.493872 23298 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:24.494074 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23169
I20260923 23:10:24.494095 23298 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:24.494158 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:24.494354 16874 external_mini_cluster.cc:1428] Setting key 038b7a9a78fe6fdd56b2cc50c6c4643d
I20260923 23:10:24.494513 22973 ts_manager.cc:194] Registered new tserver with Master: aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843)
I20260923 23:10:24.495023 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:42245
WARNING: no policy specified for oryx/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
I20260923 23:10:24.518028 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:24.597349 23303 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:24.597585 23303 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:24.597617 23303 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:24.598840 23303 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:24.598896 23303 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:24.598933 23303 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:24.600499 23303 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:39045
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23303
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:24.600761 23303 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:24.600996 23303 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:24.606079 23303 server_base.cc:1034] running on GCE node
W20260923 23:10:24.606065 23309 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.606065 23311 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.606086 23308 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.606447 23303 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.606694 23303 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.607824 23303 hybrid_clock.cc:648] HybridClock initialized: now 1790205024607812 us; error 29 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.610152 23303 init.cc:377] Logged in from keytab as oryx/127.16.122.131@KRBTEST.COM (short username oryx)
I20260923 23:10:24.610545 23303 webserver.cc:460] Webserver started at http://127.16.122.131:39195/ using document root <none> and password file <none>
I20260923 23:10:24.610752 23303 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.610791 23303 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.610882 23303 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:24.611927 23303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "c89e3f7e62424ca7ade97bcdebf4bb93"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "f061050b4899d488f9559244755b3e0a"
server_key_iv: "e6c60960964c465645db76789d2d7adc"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.612269 23303 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "c89e3f7e62424ca7ade97bcdebf4bb93"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "f061050b4899d488f9559244755b3e0a"
server_key_iv: "e6c60960964c465645db76789d2d7adc"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.613444 23303 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:24.614092 23318 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.614243 23303 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.614321 23303 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "c89e3f7e62424ca7ade97bcdebf4bb93"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "f061050b4899d488f9559244755b3e0a"
server_key_iv: "e6c60960964c465645db76789d2d7adc"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.614379 23303 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.628571 23303 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.630836 23303 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:24.630990 23303 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.631274 23303 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:24.631597 23303 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:24.631640 23303 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.631669 23303 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:24.631687 23303 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.637015 23303 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39943
I20260923 23:10:24.637053 23431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39943 every 8 connection(s)
I20260923 23:10:24.637369 23303 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:24.642643 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23303
I20260923 23:10:24.642755 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:24.642930 16874 external_mini_cluster.cc:1428] Setting key da4b2f2162b3fea2d37fb86e5f711420
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:24.646032 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.638108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:34575 (local address 127.16.122.190:39045)
0923 23:10:24.638169 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.638171 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.638368 (+ 197us) server_negotiation.cc:407] Connection header received
0923 23:10:24.638889 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.638891 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.638903 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.638954 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.639995 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.640487 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.641497 (+ 1010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.641695 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.641755 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.643925 (+ 2170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.643926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.643926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.643933 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.644977 (+ 1044us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.645321 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.645322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.645322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.645366 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.645549 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.645550 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.645550 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.645655 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.645698 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.645912 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.645967 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:10:24.646252 23434 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.638079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:34575)
0923 23:10:24.638252 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.638288 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.638790 (+ 502us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.639045 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.639052 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.639657 (+ 605us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.639890 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.639896 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.640572 (+ 676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.640573 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.641407 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.641410 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.641778 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.641779 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.641904 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.642384 (+ 480us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.642392 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.643852 (+ 1460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.645038 (+ 1186us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.645043 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.645045 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.645264 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.645410 (+ 146us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.645410 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.645410 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.645510 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.645696 (+ 186us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.645699 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.645814 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.645892 (+ 78us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.646018 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":86,"threads_started":1}
I20260923 23:10:24.646615 23432 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:24.646697 23432 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:24.646867 23432 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:24.647264 22973 ts_manager.cc:194] Registered new tserver with Master: c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943)
I20260923 23:10:24.647634 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:34575
I20260923 23:10:24.654536 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.122.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.189.keytab.
Entry for principal oryx/127.16.122.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.189.keytab.
I20260923 23:10:24.678336 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:46053
--webserver_interface=127.16.122.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.189.keytab
--principal=oryx/127.16.122.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39045,127.16.122.189:46053
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:24.759650 23437 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:24.759855 23437 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:24.759876 23437 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:24.761019 23437 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:24.761062 23437 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:24.761075 23437 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:24.761085 23437 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:24.761097 23437 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:24.762637 23437 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39045,127.16.122.189:46053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.189.keytab
--principal=oryx/127.16.122.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:46053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.122.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23437
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
W20260923 23:10:24.762744 23437 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.122.190:39045,127.16.122.189:46053'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260923 23:10:24.762897 23437 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:24.763134 23437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:24.767845 23445 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.767845 23442 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.768042 23437 server_base.cc:1034] running on GCE node
W20260923 23:10:24.767876 23443 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.768373 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.768659 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.769789 23437 hybrid_clock.cc:648] HybridClock initialized: now 1790205024769777 us; error 32 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.772372 23437 init.cc:377] Logged in from keytab as oryx/127.16.122.189@KRBTEST.COM (short username oryx)
I20260923 23:10:24.772866 23437 webserver.cc:460] Webserver started at http://127.16.122.189:39445/ using document root <none> and password file <none>
I20260923 23:10:24.773083 23437 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.773130 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.773253 23437 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:24.774294 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/instance:
uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "098d01741f28522568cc2f17b6d385d0"
server_key_iv: "2fda179d5ee424bd1a9b5039e0c087dd"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.774611 23437 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal/instance:
uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "098d01741f28522568cc2f17b6d385d0"
server_key_iv: "2fda179d5ee424bd1a9b5039e0c087dd"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.775911 23437 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:24.776660 23452 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.776813 23437 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:24.776894 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "098d01741f28522568cc2f17b6d385d0"
server_key_iv: "2fda179d5ee424bd1a9b5039e0c087dd"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.776954 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.790745 23437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.792773 23437 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:24.792882 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.796443 23437 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.189:46053
I20260923 23:10:24.796485 23504 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.189:46053 every 8 connection(s)
I20260923 23:10:24.796815 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
I20260923 23:10:24.797757 23505 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:24.799289 23505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:10:24.803225 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23437
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.189@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:24.808354 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.799973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53576 (local address 127.16.122.190:39045)
0923 23:10:24.800034 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.800036 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.800234 (+ 198us) server_negotiation.cc:407] Connection header received
0923 23:10:24.800674 (+ 440us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.800675 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.800685 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.800745 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.801326 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.801816 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.802741 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.802922 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.802970 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.805773 (+ 2803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.805775 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.805775 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.805783 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.807085 (+ 1302us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.807471 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.807472 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.807472 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.807526 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.807741 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.807742 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.807743 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.807884 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.807941 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.808219 (+ 278us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.808287 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:10:24.808571 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.799955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53576)
0923 23:10:24.800123 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.800159 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.800598 (+ 439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.800811 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.800815 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.801074 (+ 259us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.801264 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.801268 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.801874 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.801874 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.802673 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.802675 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.802977 (+ 302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.802978 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.803134 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.803860 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.803870 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.805677 (+ 1807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.807171 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.807179 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.807181 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.807400 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.807578 (+ 178us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.807579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.807579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.807686 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.807943 (+ 257us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.807949 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.808077 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.808171 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.808333 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":80,"threads_started":1}
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:24.809180 23505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:10:24.811856 23509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.803678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.16.122.189:46053)
0923 23:10:24.803814 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.803818 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.803836 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:10:24.803854 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.803855 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.803871 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.803930 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.804591 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.805645 (+ 1054us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.806573 (+ 928us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.806881 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.806941 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.809180 (+ 2239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.809185 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.809185 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.809191 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.810665 (+ 1474us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.811083 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.811086 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.811087 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.811147 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.811313 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.811313 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.811314 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.811479 (+ 165us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.811636 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.811680 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.811756 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":79,"threads_started":1}
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.189@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:24.815490 23507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.809434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54518)
0923 23:10:24.809585 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.809593 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.809623 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.809801 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.809802 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.809933 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.809980 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.809981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.810602 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.810603 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.811294 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.811297 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.811733 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.811734 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.811802 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.812132 (+ 330us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.812136 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.813188 (+ 1052us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.814687 (+ 1499us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.814687 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.814688 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.814830 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.814971 (+ 141us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.814971 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.814971 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.815017 (+ 46us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.815245 (+ 228us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.815245 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.815280 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.815333 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.815406 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140}
I20260923 23:10:24.815588 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.809476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54518 (local address 127.16.122.189:46053)
0923 23:10:24.809681 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.809683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.809691 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:24.809712 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.809713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.809723 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.809800 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.810043 (+ 243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.810538 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.811361 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.811666 (+ 305us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.811726 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.813257 (+ 1531us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.813258 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.813259 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.813264 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.814624 (+ 1360us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.814882 (+ 258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.814882 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.814883 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.814927 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.815101 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.815101 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.815101 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.815200 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.815246 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.815350 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.815448 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":41,"server-negotiator.queue_time_us":184,"thread_start_us":86,"threads_started":1}
I20260923 23:10:24.816761 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Bootstrap starting.
I20260923 23:10:24.817382 23505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:24.817629 23505 log.cc:824] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:24.818300 23505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: No bootstrap required, opened a new log
I20260923 23:10:24.819743 23505 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } }
I20260923 23:10:24.819868 23505 raft_consensus.cc:374] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:24.819900 23505 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12d3e8c418f245038c2c7bfe5dd3d534, State: Initialized, Role: FOLLOWER
I20260923 23:10:24.820022 23505 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } }
I20260923 23:10:24.820305 23505 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:24.820266 23512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } } }
I20260923 23:10:24.820356 23512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:10:24.821786 23523 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:24.821839 23523 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:24.823338 23437 master_runner.cc:428] Detected that this master 12d3e8c418f245038c2c7bfe5dd3d534 is joining an existing cluster
I20260923 23:10:24.823375 23437 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.122.189:46053
I20260923 23:10:24.830273 23530 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.824223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54530)
0923 23:10:24.824417 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.824425 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.824472 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.824583 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.824584 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.824822 (+ 238us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.824906 (+ 84us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.824908 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.825538 (+ 630us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.825539 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.826266 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.826267 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.826735 (+ 468us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.826736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.826801 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.827302 (+ 501us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.827305 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.827594 (+ 289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.829212 (+ 1618us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.829213 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.829213 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.829394 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.829589 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.829589 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.829589 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.829657 (+ 68us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.829930 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.829931 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.829973 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.830057 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.830164 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175,"thread_start_us":86,"threads_started":1}
I20260923 23:10:24.830354 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.824469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54530 (local address 127.16.122.189:46053)
0923 23:10:24.824504 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.824505 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.824511 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:10:24.824527 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.824527 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.824535 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.824573 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.824982 (+ 409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.825474 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.826343 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.826653 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.826732 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.827667 (+ 935us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.827668 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.827668 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.827674 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.829138 (+ 1464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.829463 (+ 325us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.829464 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.829465 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.829523 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.829716 (+ 193us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.829717 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.829717 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.829859 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.829953 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.830081 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.830236 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":22}
I20260923 23:10:24.830778 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.824264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53584 (local address 127.16.122.190:39045)
0923 23:10:24.824315 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.824317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.824439 (+ 122us) server_negotiation.cc:407] Connection header received
0923 23:10:24.824657 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.824658 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.824669 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.824716 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.825040 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.825785 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.826623 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.826873 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.826938 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.828231 (+ 1293us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.828232 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.828232 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.828239 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.829718 (+ 1479us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.830021 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.830022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.830023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.830080 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.830263 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.830264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.830264 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.830390 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.830432 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.830612 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.830693 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:24.831112 23529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.824215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53584)
0923 23:10:24.824379 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.824388 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.824462 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.824724 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.824725 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.824890 (+ 165us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.824942 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.824943 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.825865 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.825865 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.826552 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.826554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.827296 (+ 742us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.827297 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.827365 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.827747 (+ 382us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.827751 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.828134 (+ 383us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.829776 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.829777 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.829778 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.829963 (+ 185us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.830139 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.830140 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.830140 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.830193 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.830476 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.830476 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.830514 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.830890 (+ 376us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.830984 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":90,"threads_started":1}
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.190@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:24.839803 23510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.833148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54544 (local address 127.16.122.189:46053)
0923 23:10:24.833183 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.833184 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.833340 (+ 156us) server_negotiation.cc:407] Connection header received
0923 23:10:24.833431 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.833431 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.833439 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.833480 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.833936 (+ 456us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.834366 (+ 430us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.835439 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.835606 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.835650 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.837589 (+ 1939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.837590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.837590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.837596 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.838713 (+ 1117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.839163 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.839164 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.839164 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.839208 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.839378 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.839378 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.839378 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.839464 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.839505 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.839690 (+ 185us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.839741 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:24.839893 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.833076 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54544)
0923 23:10:24.833241 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.833281 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.833358 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.833486 (+ 128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.833489 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.833816 (+ 327us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.833873 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.833875 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.834428 (+ 553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.834429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.835369 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.835372 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.835662 (+ 290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.835662 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.835738 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.836096 (+ 358us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.836101 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.837516 (+ 1415us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.838780 (+ 1264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.838787 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.838790 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.839102 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.839265 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.839266 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.839266 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.839329 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.839512 (+ 183us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.839518 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.839575 (+ 57us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.839665 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.839782 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":84,"threads_started":1}
I20260923 23:10:24.840477 22973 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.122.189:46053: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
member_type: NON_VOTER
last_known_addr {
host: "127.16.122.189"
port: 46053
}
attrs {
promote: true
}
}
dest_uuid: "a3802b5cce894104b40cb8e29fc478fc"
cas_config_opid_index: -1
I20260923 23:10:24.840795 22973 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } }
I20260923 23:10:24.841342 23533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 12d3e8c418f245038c2c7bfe5dd3d534 (127.16.122.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } }
I20260923 23:10:24.841605 23479 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:24.842298 23479 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Refusing update from remote peer a3802b5cce894104b40cb8e29fc478fc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260923 23:10:24.842322 23533 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.122.189:46053
I20260923 23:10:24.842545 23437 master.cc:537] Master@127.16.122.189:46053 shutting down...
I20260923 23:10:24.843196 23532 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:24.844743 23437 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:24.844834 23437 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:24.844885 23437 tablet_replica.cc:330] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: stopping tablet replica
I20260923 23:10:24.844837 23534 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } }
W20260923 23:10:24.844972 22957 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
I20260923 23:10:24.845139 23534 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:24.850939 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.850811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54552)
0923 23:10:24.850863 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.850904 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:46053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
W20260923 23:10:24.851076 22957 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc -> Peer 12d3e8c418f245038c2c7bfe5dd3d534 (127.16.122.189:46053): Couldn't send request to peer 12d3e8c418f245038c2c7bfe5dd3d534. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.189:46053: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 23:10:24.857372 23437 master.cc:552] Master@127.16.122.189:46053 shutdown complete.
I20260923 23:10:24.857417 23437 master_runner.cc:305] Clearing existing system tablet
I20260923 23:10:24.857530 23437 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.857569 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.858099 23437 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.858503 23539 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.858649 23437 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.858700 23437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.858721 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "098d01741f28522568cc2f17b6d385d0"
server_key_iv: "2fda179d5ee424bd1a9b5039e0c087dd"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.859385 23437 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:24.860687 23437 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 23:10:24.860745 23437 log.cc:1196] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260923 23:10:24.860898 23437 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Deleting consensus metadata
I20260923 23:10:24.860992 23437 master_runner.cc:315] Copying system tablet from 127.16.122.190:39045
I20260923 23:10:24.861543 23437 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.190:39045
I20260923 23:10:24.867144 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.861900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53596 (local address 127.16.122.190:39045)
0923 23:10:24.861945 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.861947 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.862063 (+ 116us) server_negotiation.cc:407] Connection header received
0923 23:10:24.862129 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.862130 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.862141 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.862189 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.862513 (+ 324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.863189 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.863884 (+ 695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.864080 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.864132 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.864992 (+ 860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.864994 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.864994 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.865001 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.866125 (+ 1124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.866412 (+ 287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.866413 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.866413 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.866468 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.866629 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.866630 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.866630 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.866734 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.866787 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.866928 (+ 141us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.866999 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:10:24.867144 23545 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.861843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53596)
0923 23:10:24.861995 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:24.862004 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:24.862068 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:24.862204 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:24.862204 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:24.862385 (+ 181us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:24.862440 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.862442 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.863266 (+ 824us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.863267 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.863813 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:24.863815 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.864141 (+ 326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.864141 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:24.864204 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.864643 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:24.864647 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:24.864917 (+ 270us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:24.866195 (+ 1278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.866196 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.866196 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.866354 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.866519 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:24.866519 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:24.866520 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:24.866574 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:24.866785 (+ 211us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:24.866786 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:24.866832 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:10:24.866899 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:10:24.867000 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":81,"threads_started":1}
I20260923 23:10:24.867640 22993 tablet_copy_service.cc:140] P a3802b5cce894104b40cb8e29fc478fc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 12d3e8c418f245038c2c7bfe5dd3d534 ({username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:53596)
I20260923 23:10:24.867753 22993 tablet_copy_service.cc:161] P a3802b5cce894104b40cb8e29fc478fc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 12d3e8c418f245038c2c7bfe5dd3d534 at {username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:53596: session id = 12d3e8c418f245038c2c7bfe5dd3d534-00000000000000000000000000000000
I20260923 23:10:24.868611 22993 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 23:10:24.869231 23437 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 23:10:24.870081 23437 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: tablet copy: Starting download of 0 data blocks...
I20260923 23:10:24.870217 23437 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: tablet copy: Starting download of 1 WAL segments...
I20260923 23:10:24.871109 23437 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 23:10:24.871847 22993 tablet_copy_service.cc:342] P a3802b5cce894104b40cb8e29fc478fc: Request end of tablet copy session 12d3e8c418f245038c2c7bfe5dd3d534-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:53596
I20260923 23:10:24.871891 22993 tablet_copy_service.cc:433] P a3802b5cce894104b40cb8e29fc478fc: ending tablet copy session 12d3e8c418f245038c2c7bfe5dd3d534-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 12d3e8c418f245038c2c7bfe5dd3d534
I20260923 23:10:24.874084 23437 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:24.875176 23546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.875241 23437 server_base.cc:1034] running on GCE node
W20260923 23:10:24.875257 23547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:24.875229 23549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:24.875610 23437 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:24.875748 23437 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:24.876874 23437 hybrid_clock.cc:648] HybridClock initialized: now 1790205024876866 us; error 23 us; skew 500 ppm
Sep 23 23:10:24 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:24.878120 23437 init.cc:377] Logged in from keytab as oryx/127.16.122.189@KRBTEST.COM (short username oryx)
I20260923 23:10:24.878412 23437 webserver.cc:460] Webserver started at http://127.16.122.189:37239/ using document root <none> and password file <none>
I20260923 23:10:24.878521 23437 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:24.878554 23437 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:24.879246 23437 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.879784 23556 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:24.879894 23437 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:24.879930 23437 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "12d3e8c418f245038c2c7bfe5dd3d534"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "098d01741f28522568cc2f17b6d385d0"
server_key_iv: "2fda179d5ee424bd1a9b5039e0c087dd"
server_key_version: "encryptionkey@0"
I20260923 23:10:24.879985 23437 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:24.894927 23437 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:24.896404 23437 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:24.899237 23437 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.189:46053
I20260923 23:10:24.899463 23608 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.189:46053 every 8 connection(s)
I20260923 23:10:24.899578 23437 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
I20260923 23:10:24.900292 23609 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:10:24.900707 23609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Bootstrap starting.
I20260923 23:10:24.902794 23609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:10:24.902988 23609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Bootstrap complete.
I20260923 23:10:24.903219 23609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } }
I20260923 23:10:24.903293 23609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 12d3e8c418f245038c2c7bfe5dd3d534, State: Initialized, Role: LEARNER
I20260923 23:10:24.903319 23609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } }
I20260923 23:10:24.903471 23609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:24.903492 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } }
I20260923 23:10:24.903594 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:10:24.904769 23623 catalog_manager.cc:1259] Loaded cluster ID: e497fe61c14b41858bb5d98af5ca70d5
I20260923 23:10:24.904816 23623 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: loading cluster ID for follower catalog manager: success
I20260923 23:10:24.905876 23623 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: acquiring CA information for follower catalog manager: success
I20260923 23:10:24.906143 23623 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:10:24.920222 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:24.913366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54578 (local address 127.16.122.189:46053)
0923 23:10:24.913509 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:24.913511 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:24.913522 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:24.913549 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:24.913550 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:24.913563 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:24.913642 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:24.914133 (+ 491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.915207 (+ 1074us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.915926 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:24.916270 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:24.916346 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:24.917583 (+ 1237us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:24.917585 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:24.917585 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:24.917593 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:24.919193 (+ 1600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.919495 (+ 302us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.919496 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.919496 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.919562 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:24.919722 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:24.919723 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:24.919723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:24.919856 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:24.919950 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:24.920025 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 23:10:24.920106 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":68,"threads_started":1}
WARNING: no policy specified for oryx/127.16.122.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.188@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.188.keytab.
Entry for principal oryx/127.16.122.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.188.keytab.
I20260923 23:10:24.942978 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:45015
--webserver_interface=127.16.122.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.188.keytab
--principal=oryx/127.16.122.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:25.022413 23627 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:25.022624 23627 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:25.022645 23627 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:25.023808 23627 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:25.023850 23627 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:25.023862 23627 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:25.023873 23627 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:25.023885 23627 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:25.025314 23627 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.188.keytab
--principal=oryx/127.16.122.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:45015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.122.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23627
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:25.025507 23627 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:25.025683 23627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:25.030325 23632 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.030350 23635 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.030360 23633 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.030530 23627 server_base.cc:1034] running on GCE node
I20260923 23:10:25.030810 23627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:25.031109 23627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:25.032246 23627 hybrid_clock.cc:648] HybridClock initialized: now 1790205025032235 us; error 23 us; skew 500 ppm
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:25.034539 23627 init.cc:377] Logged in from keytab as oryx/127.16.122.188@KRBTEST.COM (short username oryx)
I20260923 23:10:25.034984 23627 webserver.cc:460] Webserver started at http://127.16.122.188:43497/ using document root <none> and password file <none>
I20260923 23:10:25.035233 23627 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.035284 23627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.035418 23627 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:25.036491 23627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/instance:
uuid: "684dee5eabbe4479ac6605d67b0d5d98"
format_stamp: "Formatted at 2026-09-23 23:10:25 on dist-test-slave-95hh"
server_key: "feb2f94326a50094dcd98da453adaed1"
server_key_iv: "bf86cc9a4fd035bba13eded3c3b88781"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.036862 23627 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal/instance:
uuid: "684dee5eabbe4479ac6605d67b0d5d98"
format_stamp: "Formatted at 2026-09-23 23:10:25 on dist-test-slave-95hh"
server_key: "feb2f94326a50094dcd98da453adaed1"
server_key_iv: "bf86cc9a4fd035bba13eded3c3b88781"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.038142 23627 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:10:25.038808 23642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.038954 23627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:25.039060 23627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "684dee5eabbe4479ac6605d67b0d5d98"
format_stamp: "Formatted at 2026-09-23 23:10:25 on dist-test-slave-95hh"
server_key: "feb2f94326a50094dcd98da453adaed1"
server_key_iv: "bf86cc9a4fd035bba13eded3c3b88781"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.039134 23627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:25.055331 23627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:25.057102 23627 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:25.057247 23627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:25.061246 23627 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.188:45015
I20260923 23:10:25.061276 23694 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.188:45015 every 8 connection(s)
I20260923 23:10:25.061616 23627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
I20260923 23:10:25.062516 23695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:25.064018 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:10:25.067270 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23627
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.188@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
I20260923 23:10:25.072698 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.064761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53610 (local address 127.16.122.190:39045)
0923 23:10:25.064823 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.064825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.064953 (+ 128us) server_negotiation.cc:407] Connection header received
0923 23:10:25.065381 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.065382 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.065391 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.065438 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.066004 (+ 566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.066476 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.067435 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.067698 (+ 263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.067769 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.069982 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.069983 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.069984 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.069991 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.071422 (+ 1431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.071779 (+ 357us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.071780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.071781 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.071830 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.072069 (+ 239us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.072070 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.072071 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.072187 (+ 116us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.072243 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.072553 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.072632 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:10:25.072907 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.064617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53610)
0923 23:10:25.064845 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.064882 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.065313 (+ 431us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.065521 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.065524 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.065759 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.065938 (+ 179us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.065942 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.066533 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.066534 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.067360 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.067363 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.067793 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.067795 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.067932 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.068394 (+ 462us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.068404 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.069906 (+ 1502us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.071499 (+ 1593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.071505 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.071507 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.071718 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.071883 (+ 165us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.071884 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.071884 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.071996 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.072250 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.072256 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.072414 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.072525 (+ 111us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.072681 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"thread_start_us":86,"threads_started":1}
I20260923 23:10:25.073403 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:10:25.075742 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.067714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50620 (local address 127.16.122.188:45015)
0923 23:10:25.067871 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.067874 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.067885 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:25.067902 (+ 17us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.067903 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.067917 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.067973 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.068659 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.069518 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.070260 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.070644 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.070717 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.073173 (+ 2456us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.073181 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.073181 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.073187 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.074575 (+ 1388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.074917 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.074919 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.074920 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.074970 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.075173 (+ 203us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.075173 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.075173 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.075295 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.075405 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.075483 (+ 78us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.075555 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":76,"threads_started":1}
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.188@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:25.079488 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.073628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54582)
0923 23:10:25.073665 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.073674 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.073706 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.073841 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.073841 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.073958 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.074001 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.074002 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.074600 (+ 598us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.074601 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.075317 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.075319 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.075799 (+ 480us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.075800 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.075880 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.076199 (+ 319us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.076203 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.077247 (+ 1044us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.078738 (+ 1491us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.078739 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.078739 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.078882 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.079063 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.079064 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.079064 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.079111 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.079292 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.079292 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.079322 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.079365 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.079428 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:10:25.079559 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.073693 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54582 (local address 127.16.122.189:46053)
0923 23:10:25.073741 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.073743 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.073751 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:25.073775 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.073776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.073789 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.073833 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.074064 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.074537 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.075396 (+ 859us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.075728 (+ 332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.075794 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.077312 (+ 1518us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.077313 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.077314 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.077320 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.078648 (+ 1328us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.078936 (+ 288us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.078937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.078937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.078992 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.079155 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.079156 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.079156 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.079250 (+ 94us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.079297 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.079382 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.079459 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:25.079902 23695 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.188@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
I20260923 23:10:25.085196 23697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.080127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50630)
0923 23:10:25.080169 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.080179 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.080219 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.080324 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.080325 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.080422 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.080448 (+ 26us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.080449 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.080995 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.080995 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.081447 (+ 452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.081448 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.081727 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.081727 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.081761 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.082035 (+ 274us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.082038 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.083136 (+ 1098us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.084429 (+ 1293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.084430 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.084430 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.084578 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.084717 (+ 139us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.084717 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.084718 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.084762 (+ 44us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.084943 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.084943 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.084978 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.085041 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.085111 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:10:25.085242 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.080184 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.16.122.188:45015)
0923 23:10:25.080218 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.080220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.080226 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:10:25.080268 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.080268 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.080276 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.080315 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.080492 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.080931 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.081501 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.081680 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.081722 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.083202 (+ 1480us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.083203 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.083204 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.083209 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.084353 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.084636 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.084637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.084637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.084676 (+ 39us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.084795 (+ 119us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.084795 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.084796 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.084876 (+ 80us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.084914 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.085060 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.085154 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":17}
I20260923 23:10:25.086625 23695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Bootstrap starting.
I20260923 23:10:25.087445 23695 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:25.087694 23695 log.cc:824] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:25.088341 23695 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: No bootstrap required, opened a new log
I20260923 23:10:25.089725 23695 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } }
I20260923 23:10:25.089850 23695 raft_consensus.cc:374] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:25.089880 23695 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 684dee5eabbe4479ac6605d67b0d5d98, State: Initialized, Role: FOLLOWER
I20260923 23:10:25.090003 23695 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } }
I20260923 23:10:25.090291 23695 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:25.090260 23700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } } }
I20260923 23:10:25.090332 23700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:10:25.091794 23711 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:25.091849 23711 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:25.093844 23627 master_runner.cc:428] Detected that this master 684dee5eabbe4479ac6605d67b0d5d98 is joining an existing cluster
I20260923 23:10:25.093888 23627 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.122.188:45015
I20260923 23:10:25.101339 23717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.094762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53620)
0923 23:10:25.094902 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.094912 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.094957 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.095225 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.095226 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.095461 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.095532 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.095534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096497 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.096498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.097448 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.097451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.097970 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097970 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.098027 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.098459 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.098464 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.098778 (+ 314us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.100439 (+ 1661us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.100440 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.100441 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.100600 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.100783 (+ 183us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.100783 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.100784 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.100829 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.101078 (+ 249us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.101079 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.101134 (+ 55us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.101188 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.101264 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"mutex_wait_us":40,"thread_start_us":75,"threads_started":1}
I20260923 23:10:25.101392 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.094948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53620 (local address 127.16.122.190:39045)
0923 23:10:25.095081 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.095082 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.095091 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:25.095114 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.095114 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.095125 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.095180 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.095623 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096404 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097544 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.097842 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097879 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.098868 (+ 989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.098870 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.098870 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.098877 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.100359 (+ 1482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.100667 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.100668 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.100669 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.100729 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.100876 (+ 147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.100877 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.100877 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.101014 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.101077 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.101223 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.101295 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205024, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.190@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
I20260923 23:10:25.126286 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.103187 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50644)
0923 23:10:25.103251 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.103260 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.103293 (+ 33us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.103514 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.103515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.103693 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.103737 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.103739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.104370 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.104371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.105221 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.105223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.105571 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.105572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.105635 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.105956 (+ 321us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.105960 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.107339 (+ 1379us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.125193 (+ 17854us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.125195 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.125195 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.125418 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.125633 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.125634 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.125634 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.125693 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.125974 (+ 281us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.125975 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.126023 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.126068 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.126186 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 23:10:25.126937 23720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.103235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.16.122.188:45015)
0923 23:10:25.103385 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.103387 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.103394 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:25.103424 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.103425 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.103439 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.103493 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.103807 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.104303 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.105290 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.105511 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.105560 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.107429 (+ 1869us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.107430 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.107430 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.107437 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.125077 (+ 17640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.125503 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.125504 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.125504 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.125559 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.125763 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.125764 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.125764 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.125902 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.126719 (+ 817us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.126755 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.126834 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":132,"thread_start_us":96,"threads_started":1}
I20260923 23:10:25.127079 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23035
I20260923 23:10:25.127323 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.095002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54584 (local address 127.16.122.189:46053)
0923 23:10:25.095067 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.095070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.095225 (+ 155us) server_negotiation.cc:407] Connection header received
0923 23:10:25.095277 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.095278 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.095288 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.095350 (+ 62us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.095695 (+ 345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096404 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097544 (+ 1140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.097845 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097880 (+ 35us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.124351 (+ 26471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.124353 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.124354 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.124364 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.126191 (+ 1827us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126526 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.126528 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.126528 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.126580 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126771 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.126772 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.126773 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.126895 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.126933 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.127168 (+ 235us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.127245 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:25.129134 22973 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.122.188:45015: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98"
member_type: NON_VOTER
last_known_addr {
host: "127.16.122.188"
port: 45015
}
attrs {
promote: true
}
}
dest_uuid: "a3802b5cce894104b40cb8e29fc478fc"
cas_config_opid_index: 5
I20260923 23:10:25.129379 22973 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } }
I20260923 23:10:25.129609 23698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.094984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.16.122.188:45015)
0923 23:10:25.095016 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.095018 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.095490 (+ 472us) server_negotiation.cc:407] Connection header received
0923 23:10:25.095568 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.095569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.095576 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.095600 (+ 24us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.095936 (+ 336us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096626 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097626 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.097895 (+ 269us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097931 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.124538 (+ 26607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.124540 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.124540 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.124546 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.126047 (+ 1501us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126371 (+ 324us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.126372 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.126372 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.126431 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.129153 (+ 2722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.129154 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.129154 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.129264 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.129299 (+ 35us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.129452 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.129523 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:10:25.129817 23532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.130838 23532 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.122.188:45015
I20260923 23:10:25.130913 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.130973 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.131220 23627 master.cc:537] Master@127.16.122.188:45015 shutting down...
I20260923 23:10:25.131505 23718 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.095069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50640)
0923 23:10:25.095435 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.095444 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.095510 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.095644 (+ 134us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.095645 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.095809 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.095863 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.095865 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096707 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.096707 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.097541 (+ 834us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.097544 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.098006 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.098007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.098060 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.123923 (+ 25863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.123928 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.124434 (+ 506us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.126130 (+ 1696us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126130 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.126131 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.126307 (+ 176us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.129025 (+ 2718us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.129026 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.129026 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.129091 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.129334 (+ 243us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.129335 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.129366 (+ 31us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.131282 (+ 1916us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.131388 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":63,"threads_started":1}
W20260923 23:10:25.131947 22958 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc -> Peer 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188:45015): Couldn't send request to peer 684dee5eabbe4479ac6605d67b0d5d98. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260923 23:10:25.133755 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.094774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54584)
0923 23:10:25.095165 (+ 391us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.095173 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.095222 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.095354 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.095355 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.095563 (+ 208us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.095620 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.095621 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.096497 (+ 876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.096498 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.097448 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.097451 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.097965 (+ 514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.097966 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.098026 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.123653 (+ 25627us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.123659 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.124212 (+ 553us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.126269 (+ 2057us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126270 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.126271 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.126450 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.126655 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.126656 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.126656 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.126707 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.126977 (+ 270us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.126978 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.127019 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.133534 (+ 6515us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.133654 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"mutex_wait_us":38,"thread_start_us":66,"threads_started":1}
I20260923 23:10:25.133790 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:41871
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:39045
--webserver_port=39773
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:10:25.136318 23627 raft_consensus.cc:2229] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:25.136411 23627 raft_consensus.cc:2258] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:25.136453 23627 tablet_replica.cc:330] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: stopping tablet replica
I20260923 23:10:25.137300 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.131232 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54590)
0923 23:10:25.132977 (+ 1745us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.132989 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.133044 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.134230 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.134231 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.134233 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.134277 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.134279 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.134976 (+ 697us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.134977 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.136164 (+ 1187us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.136167 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.137030 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.137031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.137100 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.137141 (+ 41us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.137191 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.137234 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1726}
I20260923 23:10:25.137603 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.132290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54590 (local address 127.16.122.189:46053)
0923 23:10:25.132350 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.132352 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.133030 (+ 678us) server_negotiation.cc:407] Connection header received
0923 23:10:25.133089 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.133090 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.133112 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.133173 (+ 61us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.134362 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.134900 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.136296 (+ 1396us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.136838 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.136901 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.136978 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.137496 (+ 518us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.137540 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:25.138005 23583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 LEARNER]: Refusing update from remote peer a3802b5cce894104b40cb8e29fc478fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260923 23:10:25.138211 23534 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260923 23:10:25.138768 23583 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.140142 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3802b5cce894104b40cb8e29fc478fc. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.140241 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:10:25.140278 23612 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: NON_VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: true } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.140316 23612 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:10:25.140566 23534 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: attempting to promote NON_VOTER 12d3e8c418f245038c2c7bfe5dd3d534 to VOTER
I20260923 23:10:25.140769 23534 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } }
I20260923 23:10:25.141417 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.141309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50656)
0923 23:10:25.141350 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.141378 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:10:25.141741 23583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 LEARNER]: Refusing update from remote peer a3802b5cce894104b40cb8e29fc478fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260923 23:10:25.141887 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.141812 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50672)
0923 23:10:25.141843 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.141863 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 23:10:25.141952 23533 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
W20260923 23:10:25.142164 22958 consensus_peers.cc:487] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc -> Peer 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188:45015): Couldn't send request to peer 684dee5eabbe4479ac6605d67b0d5d98. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.188:45015: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 23:10:25.142656 23532 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 12d3e8c418f245038c2c7bfe5dd3d534 (127.16.122.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.142784 23583 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 12d3e8c418f245038c2c7bfe5dd3d534 (127.16.122.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.143720 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.143797 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.143847 23532 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.143885 23532 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.143977 23719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.143844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50686)
0923 23:10:25.143912 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.143944 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:45015: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 23:10:25.144232 23723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.144322 23723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:10:25.152148 23627 master.cc:552] Master@127.16.122.188:45015 shutdown complete.
I20260923 23:10:25.152201 23627 master_runner.cc:305] Clearing existing system tablet
I20260923 23:10:25.152323 23627 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.152366 23627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.153100 23627 fs_manager.cc:709] Time spent opening directory manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.153606 23725 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.153740 23627 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:25.153798 23627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.153820 23627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "684dee5eabbe4479ac6605d67b0d5d98"
format_stamp: "Formatted at 2026-09-23 23:10:25 on dist-test-slave-95hh"
server_key: "feb2f94326a50094dcd98da453adaed1"
server_key_iv: "bf86cc9a4fd035bba13eded3c3b88781"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.154559 23627 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:25.156039 23627 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260923 23:10:25.156108 23627 log.cc:1196] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260923 23:10:25.156275 23627 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Deleting consensus metadata
I20260923 23:10:25.156373 23627 master_runner.cc:315] Copying system tablet from 127.16.122.190:39045
I20260923 23:10:25.156946 23627 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.190:39045
I20260923 23:10:25.164106 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.157311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53626 (local address 127.16.122.190:39045)
0923 23:10:25.157357 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.157359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.157434 (+ 75us) server_negotiation.cc:407] Connection header received
0923 23:10:25.157533 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.157534 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.157545 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.157598 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.157985 (+ 387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.158726 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.159714 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.159993 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.160060 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.161110 (+ 1050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.161111 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.161112 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.161119 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.162730 (+ 1611us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.163322 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.163323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.163324 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.163386 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.163591 (+ 205us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.163592 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.163593 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.163728 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.163790 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.163937 (+ 147us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.164013 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:10:25.164121 23731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.157246 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:53626)
0923 23:10:25.157374 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.157385 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.157456 (+ 71us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.157670 (+ 214us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.157671 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.157855 (+ 184us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.157911 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.157914 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.158817 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.158818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.159634 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.159637 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.160073 (+ 436us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.160074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.160142 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.160619 (+ 477us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.160624 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.161016 (+ 392us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.162822 (+ 1806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.162823 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.162823 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.163235 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.163458 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.163459 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.163459 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.163533 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.163786 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.163787 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.163839 (+ 52us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.163906 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.164019 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":76,"threads_started":1}
I20260923 23:10:25.164456 22993 tablet_copy_service.cc:140] P a3802b5cce894104b40cb8e29fc478fc: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 684dee5eabbe4479ac6605d67b0d5d98 ({username='oryx', principal='oryx/127.16.122.188@KRBTEST.COM'} at 127.0.0.1:53626)
I20260923 23:10:25.164501 22993 tablet_copy_service.cc:161] P a3802b5cce894104b40cb8e29fc478fc: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 684dee5eabbe4479ac6605d67b0d5d98 at {username='oryx', principal='oryx/127.16.122.188@KRBTEST.COM'} at 127.0.0.1:53626: session id = 684dee5eabbe4479ac6605d67b0d5d98-00000000000000000000000000000000
I20260923 23:10:25.165050 22993 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 23:10:25.165594 23627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260923 23:10:25.166898 23627 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: tablet copy: Starting download of 0 data blocks...
I20260923 23:10:25.167086 23627 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: tablet copy: Starting download of 1 WAL segments...
I20260923 23:10:25.167994 23627 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 23:10:25.168926 22993 tablet_copy_service.cc:342] P a3802b5cce894104b40cb8e29fc478fc: Request end of tablet copy session 684dee5eabbe4479ac6605d67b0d5d98-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.122.188@KRBTEST.COM'} at 127.0.0.1:53626
I20260923 23:10:25.168985 22993 tablet_copy_service.cc:433] P a3802b5cce894104b40cb8e29fc478fc: ending tablet copy session 684dee5eabbe4479ac6605d67b0d5d98-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 684dee5eabbe4479ac6605d67b0d5d98
I20260923 23:10:25.174474 23627 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:25.175734 23735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.175762 23733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.176013 23732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.180020 23627 server_base.cc:1034] running on GCE node
I20260923 23:10:25.180147 23627 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:25.180302 23627 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:25.181425 23627 hybrid_clock.cc:648] HybridClock initialized: now 1790205025181416 us; error 28 us; skew 500 ppm
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:25.182920 23627 init.cc:377] Logged in from keytab as oryx/127.16.122.188@KRBTEST.COM (short username oryx)
I20260923 23:10:25.183255 23627 webserver.cc:460] Webserver started at http://127.16.122.188:33733/ using document root <none> and password file <none>
I20260923 23:10:25.183367 23627 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.183398 23627 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.184144 23627 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:25.184705 23742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.184872 23627 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:25.184921 23627 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "684dee5eabbe4479ac6605d67b0d5d98"
format_stamp: "Formatted at 2026-09-23 23:10:25 on dist-test-slave-95hh"
server_key: "feb2f94326a50094dcd98da453adaed1"
server_key_iv: "bf86cc9a4fd035bba13eded3c3b88781"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.184975 23627 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260923 23:10:25.239696 23721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:25.239962 23721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:25.239995 23721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:25.241539 23721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:25.241600 23721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:25.241637 23721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:25.242583 23627 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:25.243755 23721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:41871
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=39773
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23721
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:25.244014 23721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:25.244036 23627 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:25.244277 23721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:25.246927 23627 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.188:45015
I20260923 23:10:25.247362 23627 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
I20260923 23:10:25.247790 23802 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.188:45015 every 8 connection(s)
I20260923 23:10:25.248313 23803 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:10:25.248749 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Bootstrap starting.
W20260923 23:10:25.249874 23785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.249874 23780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.249986 23778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.250111 23721 server_base.cc:1034] running on GCE node
I20260923 23:10:25.250306 23721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:25.250533 23721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:25.251699 23721 hybrid_clock.cc:648] HybridClock initialized: now 1790205025251684 us; error 27 us; skew 500 ppm
I20260923 23:10:25.251773 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:10:25.251924 23803 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: Bootstrap complete.
I20260923 23:10:25.252091 23803 raft_consensus.cc:348] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } }
I20260923 23:10:25.252163 23803 raft_consensus.cc:729] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 684dee5eabbe4479ac6605d67b0d5d98, State: Initialized, Role: LEARNER
I20260923 23:10:25.252190 23803 consensus_queue.cc:260] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } }
I20260923 23:10:25.252367 23803 sys_catalog.cc:564] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:25.252370 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.252512 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: LEARNER
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:25.253643 23819 catalog_manager.cc:1259] Loaded cluster ID: e497fe61c14b41858bb5d98af5ca70d5
I20260923 23:10:25.254251 23819 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: loading cluster ID for follower catalog manager: success
I20260923 23:10:25.254299 23721 init.cc:377] Logged in from keytab as oryx/127.16.122.129@KRBTEST.COM (short username oryx)
I20260923 23:10:25.254845 23721 webserver.cc:460] Webserver started at http://127.16.122.129:39773/ using document root <none> and password file <none>
I20260923 23:10:25.255121 23721 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.255144 23819 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: acquiring CA information for follower catalog manager: success
I20260923 23:10:25.255185 23721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.255353 23819 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:10:25.256491 23721 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:25.256989 23824 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.257135 23721 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:25.257211 23721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "6912c866b06549f2b43547471502e863"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "38ac62004b4d06f5f5d62dc1da721e5b"
server_key_iv: "7538b507ea510076c42afebda586222c"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.257560 23721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:25.274186 23721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:25.276098 23721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:25.276233 23721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:25.276439 23721 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:25.276727 23721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:25.276758 23721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.276785 23721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:25.276808 23721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.282505 23721 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:41871
I20260923 23:10:25.282543 23937 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:41871 every 8 connection(s)
I20260923 23:10:25.282873 23721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:25.289651 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23721
I20260923 23:10:25.289781 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23169
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:25.293494 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.283582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:60153 (local address 127.16.122.189:46053)
0923 23:10:25.283630 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.283632 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.283896 (+ 264us) server_negotiation.cc:407] Connection header received
0923 23:10:25.284542 (+ 646us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.284543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.284555 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.284583 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.285416 (+ 833us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286023 (+ 607us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287432 (+ 1409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287709 (+ 277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287763 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.290456 (+ 2693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.290458 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.290458 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.290466 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.291972 (+ 1506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292439 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.292440 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.292440 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.292495 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292765 (+ 270us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.292767 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.292767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.292911 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.292953 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.293332 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.293404 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:10:25.293632 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.284024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:38131 (local address 127.16.122.190:39045)
0923 23:10:25.284070 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.284072 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.284081 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:25.284467 (+ 386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.284468 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.284480 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.284529 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.286108 (+ 1579us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286549 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287352 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287643 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287706 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.291122 (+ 3416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.291124 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.291124 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.291132 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.292592 (+ 1460us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292927 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.292928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.292928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.292978 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293141 (+ 163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.293142 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.293142 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.293254 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.293290 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.293465 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.293548 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:25.294062 23944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.283582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.16.122.129:60153)
0923 23:10:25.283756 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.283805 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.284376 (+ 571us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.284657 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.284657 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.285123 (+ 466us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.285329 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.285335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286116 (+ 781us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.286117 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287353 (+ 1236us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.287356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287806 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287807 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287863 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.288313 (+ 450us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.288323 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.289795 (+ 1472us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.292055 (+ 2260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292063 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.292065 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.292352 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.292565 (+ 213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292566 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.292566 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.292693 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.293000 (+ 307us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.293006 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.293176 (+ 170us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.293627 (+ 451us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.293842 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":32,"thread_start_us":92,"threads_started":1}
I20260923 23:10:25.294384 23945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.283863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:38131)
0923 23:10:25.284032 (+ 169us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.284040 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.284384 (+ 344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.284912 (+ 528us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.284913 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.285133 (+ 220us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.285329 (+ 196us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.285333 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286616 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.286617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287280 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.287282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287719 (+ 437us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287720 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287855 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.288524 (+ 669us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.288528 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.290992 (+ 2464us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.292676 (+ 1684us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.292677 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.292678 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.292859 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.293038 (+ 179us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293039 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.293039 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.293088 (+ 49us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.293330 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.293330 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.293369 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.294018 (+ 649us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.294291 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":157,"thread_start_us":56,"threads_started":1}
I20260923 23:10:25.294466 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.283610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:33257 (local address 127.16.122.188:45015)
0923 23:10:25.283719 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.283721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.283896 (+ 175us) server_negotiation.cc:407] Connection header received
0923 23:10:25.284498 (+ 602us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.284500 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.284513 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.284557 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.285435 (+ 878us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286165 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287455 (+ 1290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287676 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287734 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.291146 (+ 3412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.291148 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.291148 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.291155 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.293364 (+ 2209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293720 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.293721 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.293721 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.293774 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293952 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.293953 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.293953 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.294067 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.294106 (+ 39us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.294297 (+ 191us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.294376 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":58,"threads_started":1}
I20260923 23:10:25.294551 23940 heartbeater.cc:347] Connected to a master server at 127.16.122.189:46053
I20260923 23:10:25.294874 23943 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.283582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.16.122.129:33257)
0923 23:10:25.283759 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.283805 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.284384 (+ 579us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.284648 (+ 264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.284654 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.285137 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.285329 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.285335 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.286249 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.286249 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287353 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.287356 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.287783 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.287783 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.287855 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.288573 (+ 718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.288578 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.291026 (+ 2448us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.293441 (+ 2415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293442 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.293442 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.293641 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.293840 (+ 199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.293840 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.293841 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.293889 (+ 48us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.294151 (+ 262us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.294151 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.294190 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.294537 (+ 347us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.294637 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":155,"mutex_wait_us":48,"spinlock_wait_cycles":384,"thread_start_us":94,"threads_started":1}
I20260923 23:10:25.295027 23940 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.295107 23939 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:25.295145 23939 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.295257 23940 heartbeater.cc:511] Master 127.16.122.189:46053 requested a full tablet report, sending...
I20260923 23:10:25.295257 23939 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:25.295706 22973 ts_manager.cc:194] Re-registered known tserver with Master: 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:25.295832 23573 ts_manager.cc:194] Registered new tserver with Master: 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:25.296162 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:38131
I20260923 23:10:25.296450 23938 heartbeater.cc:347] Connected to a master server at 127.16.122.188:45015
I20260923 23:10:25.296500 23938 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.296551 23938 heartbeater.cc:511] Master 127.16.122.188:45015 requested a full tablet report, sending...
I20260923 23:10:25.296896 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:44843
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:39045
--webserver_port=46429
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:10:25.296916 23762 ts_manager.cc:194] Registered new tserver with Master: 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
W20260923 23:10:25.376049 23946 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:25.376261 23946 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:25.376291 23946 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:25.377437 23946 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:25.377480 23946 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:25.377514 23946 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:25.379063 23946 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:44843
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=46429
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.23946
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:25.379297 23946 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:25.379532 23946 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:25.384284 23954 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.384297 23952 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.384294 23951 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.384384 23946 server_base.cc:1034] running on GCE node
I20260923 23:10:25.384585 23946 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:25.384835 23946 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:25.385985 23946 hybrid_clock.cc:648] HybridClock initialized: now 1790205025385971 us; error 30 us; skew 500 ppm
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:25.388294 23946 init.cc:377] Logged in from keytab as oryx/127.16.122.130@KRBTEST.COM (short username oryx)
I20260923 23:10:25.388708 23946 webserver.cc:460] Webserver started at http://127.16.122.130:46429/ using document root <none> and password file <none>
I20260923 23:10:25.388912 23946 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.388973 23946 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.390223 23946 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:25.390730 23961 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.390859 23946 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:25.390940 23946 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "aaa84a1d27d04e4596a3768e2953b51b"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "29a150b052d445f77c98e67aecee4e17"
server_key_iv: "e91688b7cc4054436cf626cdc4ccc833"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.391297 23946 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:25.422469 23946 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:25.424212 23946 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:25.424340 23946 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:25.424559 23946 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:25.424840 23946 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:25.424872 23946 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.424901 23946 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:25.424925 23946 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.430061 23946 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:44843
I20260923 23:10:25.430121 24074 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:44843 every 8 connection(s)
I20260923 23:10:25.430419 23946 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:25.431445 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 23946
I20260923 23:10:25.431589 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23303
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
I20260923 23:10:25.437486 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:39943
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:39045
--webserver_port=39195
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:36945
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
I20260923 23:10:25.440447 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431149 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:43329 (local address 127.16.122.189:46053)
0923 23:10:25.431208 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.431210 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.431396 (+ 186us) server_negotiation.cc:407] Connection header received
0923 23:10:25.432554 (+ 1158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.432555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.432567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.432596 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.433565 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.434105 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.435082 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.435282 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.435338 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.437338 (+ 2000us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.437339 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.437340 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.437347 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.438798 (+ 1451us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.439445 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.439446 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.439446 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.439501 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.439780 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.439782 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.439782 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.439913 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.439975 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.440285 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.440361 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:25.440610 24080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.16.122.130:43329)
0923 23:10:25.431278 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.431318 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.432437 (+ 1119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.432772 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.432776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.433230 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.433467 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.433473 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.434203 (+ 730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.434203 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.434984 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.434986 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.435348 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.435349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.435477 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.435946 (+ 469us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.435953 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.437240 (+ 1287us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.439090 (+ 1850us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.439096 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.439099 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.439368 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.439573 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.439574 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.439574 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.439712 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.439979 (+ 267us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.439984 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.440136 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.440241 (+ 105us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.440389 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":145,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260923 23:10:25.441010 24077 heartbeater.cc:347] Connected to a master server at 127.16.122.189:46053
I20260923 23:10:25.441118 24077 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.441586 24077 heartbeater.cc:511] Master 127.16.122.189:46053 requested a full tablet report, sending...
I20260923 23:10:25.442116 23573 ts_manager.cc:194] Registered new tserver with Master: aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843)
I20260923 23:10:25.443094 24081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431362 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:40865)
0923 23:10:25.431585 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.431593 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.432451 (+ 858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.432772 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.432776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.433354 (+ 578us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.433467 (+ 113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.433472 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.436456 (+ 2984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.436457 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.437171 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.437173 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.437862 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.437862 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.437921 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.438358 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.438362 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.440245 (+ 1883us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.442020 (+ 1775us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.442021 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.442021 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.442211 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.442420 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.442421 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.442421 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.442472 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.442768 (+ 296us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.442769 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.442809 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.442874 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.442961 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":212,"thread_start_us":50,"threads_started":1}
I20260923 23:10:25.443131 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:40865 (local address 127.16.122.190:39045)
0923 23:10:25.431538 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.431540 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.431650 (+ 110us) server_negotiation.cc:407] Connection header received
0923 23:10:25.432558 (+ 908us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.432560 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.432573 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.432602 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.435406 (+ 2804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.436365 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.437250 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.437518 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.437586 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.440334 (+ 2748us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.440336 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.440336 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.440344 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.441925 (+ 1581us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.442293 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.442294 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.442295 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.442352 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.442556 (+ 204us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.442557 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.442557 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.442675 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.442733 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.442932 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.443012 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:25.443418 24076 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:25.443475 24076 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.443533 24076 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:25.443760 22973 ts_manager.cc:194] Re-registered known tserver with Master: aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843)
I20260923 23:10:25.444226 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:40865
I20260923 23:10:25.445147 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:47143 (local address 127.16.122.188:45015)
0923 23:10:25.431213 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.431215 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.431398 (+ 183us) server_negotiation.cc:407] Connection header received
0923 23:10:25.432554 (+ 1156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.432555 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.432567 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.432596 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.433659 (+ 1063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.434369 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.437266 (+ 2897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.437769 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.437809 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.441769 (+ 3960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.441771 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.441771 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.441778 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.443340 (+ 1562us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.444372 (+ 1032us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.444373 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.444373 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.444434 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.444621 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.444622 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.444622 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.444753 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.444818 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.444962 (+ 144us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.445047 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260923 23:10:25.445175 24079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.431096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.16.122.130:47143)
0923 23:10:25.431332 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.431342 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.432443 (+ 1101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.432772 (+ 329us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.432776 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.433179 (+ 403us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.433575 (+ 396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.433576 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.434460 (+ 884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.434461 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.435312 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.435314 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.438084 (+ 2770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.438085 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.438135 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.438888 (+ 753us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.438893 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.441618 (+ 2725us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.443416 (+ 1798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.443417 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.443418 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.443655 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.444488 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.444489 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.444489 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.444551 (+ 62us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.444815 (+ 264us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.444816 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.444852 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.444906 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.444984 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":226,"thread_start_us":86,"threads_started":1}
I20260923 23:10:25.445458 24075 heartbeater.cc:347] Connected to a master server at 127.16.122.188:45015
I20260923 23:10:25.445513 24075 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.445573 24075 heartbeater.cc:511] Master 127.16.122.188:45015 requested a full tablet report, sending...
I20260923 23:10:25.445835 23762 ts_manager.cc:194] Registered new tserver with Master: aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843)
W20260923 23:10:25.519323 24082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:25.519522 24082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:25.519546 24082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:25.520696 24082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:25.520738 24082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:25.520766 24082 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:25.522241 24082 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:36945
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:39943
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=39195
--tserver_master_addrs=127.16.122.190:39045,127.16.122.189:46053,127.16.122.188:45015
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24082
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:25.522459 24082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:25.522675 24082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:25.527287 24087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:25.527333 24088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.527488 24082 server_base.cc:1034] running on GCE node
W20260923 23:10:25.527344 24090 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:25.527729 24082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:25.527954 24082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:25.529081 24082 hybrid_clock.cc:648] HybridClock initialized: now 1790205025529072 us; error 28 us; skew 500 ppm
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:25.531450 24082 init.cc:377] Logged in from keytab as oryx/127.16.122.131@KRBTEST.COM (short username oryx)
I20260923 23:10:25.531862 24082 webserver.cc:460] Webserver started at http://127.16.122.131:39195/ using document root <none> and password file <none>
I20260923 23:10:25.532060 24082 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:25.532142 24082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:25.533376 24082 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:25.533857 24097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.534005 24082 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.534080 24082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "c89e3f7e62424ca7ade97bcdebf4bb93"
format_stamp: "Formatted at 2026-09-23 23:10:24 on dist-test-slave-95hh"
server_key: "f061050b4899d488f9559244755b3e0a"
server_key_iv: "e6c60960964c465645db76789d2d7adc"
server_key_version: "encryptionkey@0"
I20260923 23:10:25.534403 24082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:25.553864 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.550985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50688)
0923 23:10:25.551068 (+ 83us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.551079 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.551140 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.551289 (+ 149us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.551290 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.551291 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.551340 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.551342 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.552001 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.552003 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.552960 (+ 957us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.552963 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.553642 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.553643 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.553697 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.553700 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.553738 (+ 38us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.553781 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 23:10:25.553872 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.551083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50688 (local address 127.16.122.188:45015)
0923 23:10:25.551141 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.551143 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.551153 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:25.551201 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.551202 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.551226 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.551279 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.551390 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.551936 (+ 546us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.553031 (+ 1095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.553576 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.553649 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.553741 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.553758 (+ 17us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.553817 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:10:25.554302 23772 raft_consensus.cc:1203] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260923 23:10:25.554457 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a3802b5cce894104b40cb8e29fc478fc. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.554570 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: LEARNER
I20260923 23:10:25.558799 24082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:25.560601 24082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:25.560730 24082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:25.560942 24082 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:25.561235 24082 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:25.561268 24082 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.561298 24082 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:25.561318 24082 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:25.566526 24082 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39943
I20260923 23:10:25.566588 24210 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39943 every 8 connection(s)
I20260923 23:10:25.566885 24082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:25.572760 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 24082
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.188@KRBTEST.COM
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.189@KRBTEST.COM
Sep 23 23:10:25 dist-test-slave-95hh krb5kdc[22928](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205025, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:25.578389 24215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.16.122.131:55907)
0923 23:10:25.567772 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.567806 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.568756 (+ 950us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.568999 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.569003 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.569431 (+ 428us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.569624 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.569630 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.571115 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571116 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.571926 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.571928 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.572307 (+ 379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.572308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.572372 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.573269 (+ 897us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.573273 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.575100 (+ 1827us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.576774 (+ 1674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.576780 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.576783 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.577051 (+ 268us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.577246 (+ 195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.577247 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.577247 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.577376 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.577662 (+ 286us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.577669 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.577842 (+ 173us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.577956 (+ 114us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.578139 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":138,"mutex_wait_us":3,"thread_start_us":99,"threads_started":1}
I20260923 23:10:25.579188 23624 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:55907 (local address 127.16.122.189:46053)
0923 23:10:25.567624 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.567627 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.567889 (+ 262us) server_negotiation.cc:407] Connection header received
0923 23:10:25.568863 (+ 974us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.568865 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.568877 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.568926 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.570288 (+ 1362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.570994 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571992 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.572245 (+ 253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.572306 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.575220 (+ 2914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.575221 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.575221 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.575227 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.576704 (+ 1477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.577119 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.577120 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.577120 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.577179 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.577440 (+ 261us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.577441 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.577441 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.577566 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.577625 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.579016 (+ 1391us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.579112 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:25.579563 24213 heartbeater.cc:347] Connected to a master server at 127.16.122.189:46053
I20260923 23:10:25.579680 24213 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.579895 24213 heartbeater.cc:511] Master 127.16.122.189:46053 requested a full tablet report, sending...
I20260923 23:10:25.580164 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567597 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:59201)
0923 23:10:25.568040 (+ 443us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.568048 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.568770 (+ 722us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.568998 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.569003 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.569440 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.569624 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.569636 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.571931 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571932 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.572613 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.572615 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.573086 (+ 471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.573087 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.573156 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.573511 (+ 355us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.573515 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.575771 (+ 2256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.577923 (+ 2152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.577924 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.577925 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.578914 (+ 989us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.579506 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.579507 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.579507 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.579570 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.579862 (+ 292us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.579863 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.579903 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.579961 (+ 58us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.580056 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":411,"mutex_wait_us":14,"spinlock_wait_cycles":14720,"thread_start_us":99,"threads_started":1}
I20260923 23:10:25.580365 23573 ts_manager.cc:194] Registered new tserver with Master: c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943)
I20260923 23:10:25.580451 23032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:59201 (local address 127.16.122.190:39045)
0923 23:10:25.567681 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.567683 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.568104 (+ 421us) server_negotiation.cc:407] Connection header received
0923 23:10:25.568853 (+ 749us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.568854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.568868 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.568923 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.571100 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.571855 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.572730 (+ 875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.573006 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.573077 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.575854 (+ 2777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.575855 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.575855 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.575862 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.577836 (+ 1974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.579369 (+ 1533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.579370 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.579371 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.579432 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.579648 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.579649 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.579650 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.579775 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.579836 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.580282 (+ 446us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.580357 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:10:25.580902 24212 heartbeater.cc:347] Connected to a master server at 127.16.122.190:39045
I20260923 23:10:25.580964 24212 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.581035 24212 heartbeater.cc:511] Master 127.16.122.190:39045 requested a full tablet report, sending...
I20260923 23:10:25.581279 22973 ts_manager.cc:194] Re-registered known tserver with Master: c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943)
I20260923 23:10:25.581739 22973 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:59201
I20260923 23:10:25.582412 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.573960 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54600 (local address 127.16.122.189:46053)
0923 23:10:25.574058 (+ 98us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.574060 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.574067 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:25.574086 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.574086 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.574094 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.574929 (+ 835us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.574965 (+ 36us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.575653 (+ 688us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.576666 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.576940 (+ 274us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.577247 (+ 307us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.578314 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.578315 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.578315 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.578322 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.580733 (+ 2411us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.581407 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.581408 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.581409 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.581471 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.581660 (+ 189us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.581661 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.581661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.581780 (+ 119us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.582213 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.582250 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.582324 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81,"thread_start_us":46,"threads_started":1}
I20260923 23:10:25.583562 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567950 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:52053 (local address 127.16.122.188:45015)
0923 23:10:25.567992 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.567994 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.568967 (+ 973us) server_negotiation.cc:407] Connection header received
0923 23:10:25.569037 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.569038 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.569048 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.569092 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.569714 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.570200 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571480 (+ 1280us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.571668 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571721 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.578416 (+ 6695us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.578417 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.578417 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.578425 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.580604 (+ 2179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.582757 (+ 2153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.582758 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.582759 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.582815 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.582990 (+ 175us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.582991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.582991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.583151 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.583224 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.583384 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.583467 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:25.583777 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.567887 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.16.122.131:52053)
0923 23:10:25.568915 (+ 1028us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.568924 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.568989 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.569095 (+ 106us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.569096 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.569417 (+ 321us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:25.569623 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.569628 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.570293 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.570294 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.571408 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.571411 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.571733 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.571733 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.571847 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.572579 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:25.572588 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:25.578323 (+ 5735us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:25.582483 (+ 4160us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.582484 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.582484 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.582675 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.582883 (+ 208us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:25.582883 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:25.582883 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:25.582939 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:25.583229 (+ 290us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:25.583230 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:25.583273 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.583346 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.583456 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1012,"thread_start_us":54,"threads_started":1}
I20260923 23:10:25.584081 24211 heartbeater.cc:347] Connected to a master server at 127.16.122.188:45015
I20260923 23:10:25.584136 24211 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:25.584146 24225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.574228 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53630 (local address 127.16.122.190:39045)
0923 23:10:25.574765 (+ 537us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.574767 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.574774 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:25.574793 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.574793 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.574804 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.575530 (+ 726us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.575563 (+ 33us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.577119 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.578136 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.578429 (+ 293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.578713 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.581116 (+ 2403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.581117 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.581117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.581123 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.582746 (+ 1623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.583397 (+ 651us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.583399 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.583399 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.583453 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.583641 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.583642 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.583642 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.583760 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.583851 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.583979 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.584060 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":79,"threads_started":1}
I20260923 23:10:25.584292 24211 heartbeater.cc:511] Master 127.16.122.188:45015 requested a full tablet report, sending...
I20260923 23:10:25.584579 23762 ts_manager.cc:194] Registered new tserver with Master: c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943)
I20260923 23:10:25.586756 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.574445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50690 (local address 127.16.122.188:45015)
0923 23:10:25.574545 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.574546 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.575486 (+ 940us) server_negotiation.cc:407] Connection header received
0923 23:10:25.575508 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.575509 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.575520 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.575598 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.580579 (+ 4981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.581282 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.582200 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.582485 (+ 285us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.582544 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.584412 (+ 1868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.584414 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.584414 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.584421 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.585817 (+ 1396us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.586106 (+ 289us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.586107 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.586108 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.586167 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.586321 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.586321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.586322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.586420 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.586499 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.586588 (+ 89us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.586658 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84,"thread_start_us":55,"threads_started":1}
I20260923 23:10:25.585736 22973 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53630:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:25.587404 22973 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:25.588033 23534 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: NON_VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: true } } }
I20260923 23:10:25.588117 23534 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.589270 23533 raft_consensus.cc:1050] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: attempting to promote NON_VOTER 684dee5eabbe4479ac6605d67b0d5d98 to VOTER
I20260923 23:10:25.589389 23533 consensus_queue.cc:237] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } }
I20260923 23:10:25.589818 23583 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Refusing update from remote peer a3802b5cce894104b40cb8e29fc478fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 23:10:25.589987 23535 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 23:10:25.590131 23772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 LEARNER]: Refusing update from remote peer a3802b5cce894104b40cb8e29fc478fc: Log matching property violated. Preceding OpId in replica: term: 1 index: 8. Preceding OpId from leader: term: 1 index: 9. (index mismatch)
I20260923 23:10:25.590333 23535 consensus_queue.cc:1035] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [LEADER]: Connected to new peer: Peer: permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 9, Last known committed idx: 8, Time since last communication: 0.000s
I20260923 23:10:25.590616 23533 raft_consensus.cc:2932] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [term 1 LEADER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.590617 23583 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591092 23772 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 FOLLOWER]: Committing config change with OpId 1.9: config changed from index 7 to 9, 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188) changed from NON_VOTER to VOTER. New config: { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591457 23533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591530 23533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.591583 23533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591581 23723 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591634 23533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.591650 23723 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:10:25.591674 23533 sys_catalog.cc:455] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.591722 23533 sys_catalog.cc:458] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:25.592118 23808 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a3802b5cce894104b40cb8e29fc478fc" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:25.592197 23808 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:10:25.596951 24238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.593055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41871 (local address 127.0.0.1:48492)
0923 23:10:25.593169 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.593178 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.593216 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.593525 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.593526 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.593528 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.593562 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.593564 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.594810 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.594811 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.595954 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.595956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.596717 (+ 761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.596718 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.596776 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.596780 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.596833 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.596874 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":101,"mutex_wait_us":9,"thread_start_us":75,"threads_started":1}
I20260923 23:10:25.596962 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.593163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48492 (local address 127.16.122.129:41871)
0923 23:10:25.593335 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.593339 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.593350 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:25.593379 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.593381 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.593393 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.593464 (+ 71us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.593617 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.594751 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.596052 (+ 1301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.596645 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.596725 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.596786 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.596848 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.596885 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":70,"threads_started":1}
I20260923 23:10:25.597043 23531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.592787 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39943 (local address 127.0.0.1:55212)
0923 23:10:25.592837 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.592848 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.592888 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.593362 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.593363 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.593365 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.593413 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.593415 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.594699 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.594700 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.595954 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.595956 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.596855 (+ 899us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.596856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.596909 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.596912 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.596939 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.596976 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33,"mutex_wait_us":24}
I20260923 23:10:25.597577 23872 tablet_service.cc:1467] Processing CreateTablet for tablet 15eafc9d46614a41b88eafae43fa613e (DEFAULT_TABLE table=test-table [id=2f5865ad4de24e1095e02f328944d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:25.597723 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.592882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55212 (local address 127.16.122.131:39943)
0923 23:10:25.593194 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.593199 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.593208 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:25.593247 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.593249 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.593264 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.593462 (+ 198us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.593516 (+ 54us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.594633 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.596036 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.596761 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.597520 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.597580 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.597604 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.597644 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":97,"threads_started":1}
I20260923 23:10:25.597903 23872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15eafc9d46614a41b88eafae43fa613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:25.598037 24235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.592903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:44843 (local address 127.0.0.1:38336)
0923 23:10:25.593024 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.593033 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.593075 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.594919 (+ 1844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.594920 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.594922 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.594980 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.594981 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.595789 (+ 808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.595790 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.596828 (+ 1038us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.596830 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.597846 (+ 1016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.597847 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.597906 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.597910 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.597938 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.597975 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":74,"threads_started":1}
I20260923 23:10:25.598312 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.592985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38336 (local address 127.16.122.130:44843)
0923 23:10:25.594775 (+ 1790us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.594779 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.594787 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:25.594822 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.594824 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.594838 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.594903 (+ 65us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.595053 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.595718 (+ 665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.597049 (+ 1331us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.597755 (+ 706us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.598140 (+ 385us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.598202 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.598226 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.598256 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1755,"thread_start_us":88,"threads_started":1}
I20260923 23:10:25.598340 24145 tablet_service.cc:1467] Processing CreateTablet for tablet 15eafc9d46614a41b88eafae43fa613e (DEFAULT_TABLE table=test-table [id=2f5865ad4de24e1095e02f328944d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:25.598650 24145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15eafc9d46614a41b88eafae43fa613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:25.598821 24009 tablet_service.cc:1467] Processing CreateTablet for tablet 15eafc9d46614a41b88eafae43fa613e (DEFAULT_TABLE table=test-table [id=2f5865ad4de24e1095e02f328944d904]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:25.599097 24009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 15eafc9d46614a41b88eafae43fa613e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:25.601430 24240 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Bootstrap starting.
I20260923 23:10:25.601894 24242 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Bootstrap starting.
I20260923 23:10:25.602279 24240 tablet_bootstrap.cc:654] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:25.602433 24242 tablet_bootstrap.cc:654] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:25.602591 24240 log.cc:824] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:25.602723 24242 log.cc:824] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:25.602916 24241 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Bootstrap starting.
I20260923 23:10:25.603365 24240 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: No bootstrap required, opened a new log
I20260923 23:10:25.603425 24240 ts_tablet_manager.cc:1397] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:25.603606 24241 tablet_bootstrap.cc:654] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:25.603621 24242 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: No bootstrap required, opened a new log
I20260923 23:10:25.603681 24242 ts_tablet_manager.cc:1397] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:25.603947 24241 log.cc:824] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:25.604698 24241 tablet_bootstrap.cc:492] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: No bootstrap required, opened a new log
I20260923 23:10:25.604765 24241 ts_tablet_manager.cc:1397] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:25.605640 24242 raft_consensus.cc:348] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.605755 24242 raft_consensus.cc:374] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:25.605774 24242 raft_consensus.cc:729] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89e3f7e62424ca7ade97bcdebf4bb93, State: Initialized, Role: FOLLOWER
I20260923 23:10:25.605695 24240 raft_consensus.cc:348] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.605836 24240 raft_consensus.cc:374] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:25.605865 24240 raft_consensus.cc:729] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6912c866b06549f2b43547471502e863, State: Initialized, Role: FOLLOWER
I20260923 23:10:25.605902 24242 consensus_queue.cc:260] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.605975 24240 consensus_queue.cc:260] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.606122 24242 ts_tablet_manager.cc:1428] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:10:25.606225 24240 ts_tablet_manager.cc:1428] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:10:25.606308 24212 heartbeater.cc:503] Master 127.16.122.190:39045 was elected leader, sending a full tablet report...
I20260923 23:10:25.606820 23939 heartbeater.cc:503] Master 127.16.122.190:39045 was elected leader, sending a full tablet report...
I20260923 23:10:25.607052 24241 raft_consensus.cc:348] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.607203 24241 raft_consensus.cc:374] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:25.607239 24241 raft_consensus.cc:729] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa84a1d27d04e4596a3768e2953b51b, State: Initialized, Role: FOLLOWER
I20260923 23:10:25.607331 24241 consensus_queue.cc:260] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.607543 24241 ts_tablet_manager.cc:1428] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:10:25.607661 24076 heartbeater.cc:503] Master 127.16.122.190:39045 was elected leader, sending a full tablet report...
I20260923 23:10:25.662341 24246 raft_consensus.cc:482] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:25.662492 24246 raft_consensus.cc:504] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.662838 24246 leader_election.cc:288] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843), 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:25.666417 24239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.663180 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36147 (local address 127.16.122.129:41871)
0923 23:10:25.663229 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.663232 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.663239 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:25.663263 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.663264 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.663272 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.663321 (+ 49us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.663421 (+ 100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.664158 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.665566 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.666103 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.666174 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.666181 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.666317 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.666352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:10:25.666419 24237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.663085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36221 (local address 127.16.122.130:44843)
0923 23:10:25.663130 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.663132 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.663142 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:25.663192 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.663193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.663201 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.663255 (+ 54us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:25.663411 (+ 156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.664148 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.665567 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.666112 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.666173 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.666181 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.666317 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.666352 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:25.666429 24216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.662985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:44843 (local address 127.16.122.131:36221)
0923 23:10:25.663063 (+ 78us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.663075 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.663139 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.663264 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.663265 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.663266 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.663341 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.663342 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.664230 (+ 888us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.664231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.665484 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.665486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.666178 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.666178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.666250 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.666254 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.666305 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.666340 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35,"mutex_wait_us":65}
I20260923 23:10:25.666438 24217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.663110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41871 (local address 127.16.122.131:36147)
0923 23:10:25.663170 (+ 60us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:25.663180 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:25.663216 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:25.663328 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:25.663329 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:25.663330 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:25.663366 (+ 36us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.663367 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.664230 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.664231 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.665484 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:25.665486 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.666178 (+ 692us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.666178 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:25.666249 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.666253 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:25.666305 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 23:10:25.666340 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45,"mutex_wait_us":38}
I20260923 23:10:25.666810 24029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15eafc9d46614a41b88eafae43fa613e" candidate_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa84a1d27d04e4596a3768e2953b51b" is_pre_election: true
I20260923 23:10:25.666852 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15eafc9d46614a41b88eafae43fa613e" candidate_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6912c866b06549f2b43547471502e863" is_pre_election: true
I20260923 23:10:25.666965 24029 raft_consensus.cc:2454] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89e3f7e62424ca7ade97bcdebf4bb93 in term 0.
I20260923 23:10:25.666965 23892 raft_consensus.cc:2454] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c89e3f7e62424ca7ade97bcdebf4bb93 in term 0.
I20260923 23:10:25.667196 24099 leader_election.cc:302] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6912c866b06549f2b43547471502e863, c89e3f7e62424ca7ade97bcdebf4bb93; no voters:
I20260923 23:10:25.667315 24246 raft_consensus.cc:2781] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:25.667380 24246 raft_consensus.cc:482] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:25.667404 24246 raft_consensus.cc:3037] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:25.668110 24246 raft_consensus.cc:504] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.668219 24246 leader_election.cc:288] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [CANDIDATE]: Term 1 election: Requested vote from peers aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130:44843), 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:25.668355 24029 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15eafc9d46614a41b88eafae43fa613e" candidate_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aaa84a1d27d04e4596a3768e2953b51b"
I20260923 23:10:25.668390 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "15eafc9d46614a41b88eafae43fa613e" candidate_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6912c866b06549f2b43547471502e863"
I20260923 23:10:25.668432 24029 raft_consensus.cc:3037] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:25.668447 23892 raft_consensus.cc:3037] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:25.669234 23892 raft_consensus.cc:2454] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89e3f7e62424ca7ade97bcdebf4bb93 in term 1.
I20260923 23:10:25.669234 24029 raft_consensus.cc:2454] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c89e3f7e62424ca7ade97bcdebf4bb93 in term 1.
I20260923 23:10:25.669400 24099 leader_election.cc:302] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6912c866b06549f2b43547471502e863, c89e3f7e62424ca7ade97bcdebf4bb93; no voters:
I20260923 23:10:25.669479 24246 raft_consensus.cc:2781] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:25.669616 24246 raft_consensus.cc:686] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 LEADER]: Becoming Leader. State: Replica: c89e3f7e62424ca7ade97bcdebf4bb93, State: Running, Role: LEADER
I20260923 23:10:25.669699 24246 consensus_queue.cc:237] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:25.670655 22973 catalog_manager.cc:5495] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 reported cstate change: term changed from 0 to 1, leader changed from <none> to c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } health_report { overall_health: UNKNOWN } } }
W20260923 23:10:25.681082 24078 tablet.cc:2369] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:25.686434 24236 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.684090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55226 (local address 127.16.122.131:39943)
0923 23:10:25.684139 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.684140 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.684166 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:10:25.684199 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.684200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.684217 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.684281 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:25.684410 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.684941 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.685703 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.685896 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.685950 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.686120 (+ 170us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:25.686223 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:25.686294 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.686331 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.686353 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:10:25.688242 24029 raft_consensus.cc:1261] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Refusing update from remote peer c89e3f7e62424ca7ade97bcdebf4bb93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:25.688242 23892 raft_consensus.cc:1261] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Refusing update from remote peer c89e3f7e62424ca7ade97bcdebf4bb93: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:25.688591 24249 consensus_queue.cc:1035] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:25.688717 24246 consensus_queue.cc:1035] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:25.692821 24254 mvcc.cc:205] Tried to move back new op lower bound from 7332679785216479232 to 7332679785143779328. Current Snapshot: MvccSnapshot[applied={T|T < 7332679785216479232}]
I20260923 23:10:25.693572 24253 mvcc.cc:205] Tried to move back new op lower bound from 7332679785216479232 to 7332679785143779328. Current Snapshot: MvccSnapshot[applied={T|T < 7332679785216479232}]
I20260923 23:10:25.693965 22973 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:25.694061 22973 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53630:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:25.695377 22973 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a3802b5cce894104b40cb8e29fc478fc: Sending DeleteTablet for 3 replicas of tablet 15eafc9d46614a41b88eafae43fa613e
I20260923 23:10:25.695752 23872 tablet_service.cc:1514] Processing DeleteTablet for tablet 15eafc9d46614a41b88eafae43fa613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:25 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:48492
I20260923 23:10:25.695765 24145 tablet_service.cc:1514] Processing DeleteTablet for tablet 15eafc9d46614a41b88eafae43fa613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:25 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:55212
I20260923 23:10:25.695847 24009 tablet_service.cc:1514] Processing DeleteTablet for tablet 15eafc9d46614a41b88eafae43fa613e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:25 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38336
I20260923 23:10:25.696293 24264 tablet_replica.cc:330] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: stopping tablet replica
I20260923 23:10:25.696391 24264 raft_consensus.cc:2229] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:25.696483 24264 raft_consensus.cc:2258] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:25.696630 24263 tablet_replica.cc:330] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: stopping tablet replica
I20260923 23:10:25.696723 24263 raft_consensus.cc:2229] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:25.696815 24263 raft_consensus.cc:2258] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:25.696879 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 22941
I20260923 23:10:25.697531 24264 ts_tablet_manager.cc:1905] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:25.699113 24264 ts_tablet_manager.cc:1918] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:25.699183 24264 log.cc:1196] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/15eafc9d46614a41b88eafae43fa613e
I20260923 23:10:25.699410 24264 ts_tablet_manager.cc:1939] T 15eafc9d46614a41b88eafae43fa613e P aaa84a1d27d04e4596a3768e2953b51b: Deleting consensus metadata
I20260923 23:10:25.699900 24263 ts_tablet_manager.cc:1905] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:25.701344 24263 ts_tablet_manager.cc:1918] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:25.701408 24263 log.cc:1196] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/15eafc9d46614a41b88eafae43fa613e
I20260923 23:10:25.701628 24263 ts_tablet_manager.cc:1939] T 15eafc9d46614a41b88eafae43fa613e P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting consensus metadata
W20260923 23:10:25.702953 23964 connection.cc:535] server connection from 127.0.0.1:38336 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260923 23:10:25.703153 24100 connection.cc:535] server connection from 127.0.0.1:55212 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:10:25.703891 24262 tablet_replica.cc:330] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: stopping tablet replica
I20260923 23:10:25.704007 24262 raft_consensus.cc:2229] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:25.704146 24262 raft_consensus.cc:2258] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:25.704643 24262 ts_tablet_manager.cc:1905] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:25.706173 24262 ts_tablet_manager.cc:1918] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:25.706247 24262 log.cc:1196] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/15eafc9d46614a41b88eafae43fa613e
I20260923 23:10:25.706478 24262 ts_tablet_manager.cc:1939] T 15eafc9d46614a41b88eafae43fa613e P 6912c866b06549f2b43547471502e863: Deleting consensus metadata
W20260923 23:10:25.706852 23827 connection.cc:441] server connection from 127.0.0.1:48492 torn down before Call kudu.tserver.TabletServerAdminService.DeleteTablet from 127.0.0.1:48492 (request call id 1) could send its response
I20260923 23:10:25.711867 24227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.696202 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50700 (local address 127.16.122.188:45015)
0923 23:10:25.696245 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.696247 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.703778 (+ 7531us) server_negotiation.cc:407] Connection header received
0923 23:10:25.703839 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.703840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.703854 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.703901 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.705029 (+ 1128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.705750 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.706772 (+ 1022us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.707097 (+ 325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.707155 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.708662 (+ 1507us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.708663 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.708663 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.708671 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.710646 (+ 1975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.710990 (+ 344us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.710991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.710992 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.711095 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.711283 (+ 188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.711284 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.711284 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.711409 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.711504 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.711684 (+ 180us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.711769 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:10:25.713373 24223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.704641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54608 (local address 127.16.122.189:46053)
0923 23:10:25.704679 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.704681 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.705618 (+ 937us) server_negotiation.cc:407] Connection header received
0923 23:10:25.705709 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.705710 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.705724 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.705751 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.706965 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.707769 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.708827 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.709142 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.709180 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.710482 (+ 1302us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.710483 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.710484 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.710491 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.712129 (+ 1638us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.712677 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.712678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.712678 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.712741 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.712925 (+ 184us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.712926 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.712926 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.713033 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.713119 (+ 86us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.713211 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.713285 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:25.714210 23942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:25.705319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50704 (local address 127.16.122.188:45015)
0923 23:10:25.705357 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:25.705359 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:25.707138 (+ 1779us) server_negotiation.cc:407] Connection header received
0923 23:10:25.707242 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:25.707243 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:25.707252 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:25.707279 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:25.707944 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.708651 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.709549 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:25.709848 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:25.709908 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:25.711227 (+ 1319us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:25.711228 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:25.711229 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:25.711235 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:25.713221 (+ 1986us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.713597 (+ 376us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.713598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.713598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.713648 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:25.713812 (+ 164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:25.713813 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:25.713813 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:25.713910 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:25.713978 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:25.714101 (+ 123us) server_negotiation.cc:299] Negotiation successful
0923 23:10:25.714153 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:26.692890 24273 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:26.692572 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:40171)
0923 23:10:26.692736 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:26.692831 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":102,"threads_started":1}
W20260923 23:10:26.693166 23939 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:26.694172 24274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:26.693955 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:51211)
0923 23:10:26.694052 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:26.694126 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":59,"threads_started":1}
W20260923 23:10:26.694344 24076 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:26.694694 24275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:26.694512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:33477)
0923 23:10:26.694617 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:26.694663 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
W20260923 23:10:26.694851 24212 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:39045 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.195919 24276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3802b5cce894104b40cb8e29fc478fc)
I20260923 23:10:27.196041 24276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } }
I20260923 23:10:27.196290 24276 leader_election.cc:288] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045), 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188:45015)
I20260923 23:10:27.196405 24277 raft_consensus.cc:482] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader a3802b5cce894104b40cb8e29fc478fc)
I20260923 23:10:27.196511 24277 raft_consensus.cc:504] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } }
I20260923 23:10:27.196684 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54192)
0923 23:10:27.196581 (+ 95us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.196635 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75,"thread_start_us":54,"threads_started":1}
I20260923 23:10:27.196749 24277 leader_election.cc:288] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045), 12d3e8c418f245038c2c7bfe5dd3d534 (127.16.122.189:46053)
W20260923 23:10:27.197012 23558 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.197062 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54196)
0923 23:10:27.196978 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.197024 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
W20260923 23:10:27.197235 23744 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.197484 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.197382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54204)
0923 23:10:27.197419 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.197453 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260923 23:10:27.197564 23558 leader_election.cc:334] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045): Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.197642 24282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.197563 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54214)
0923 23:10:27.197599 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.197620 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:10:27.197707 23744 leader_election.cc:334] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045): Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.200035 24281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50716 (local address 127.16.122.188:45015)
0923 23:10:27.197144 (+ 278us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.197146 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.197156 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:27.197184 (+ 28us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.197185 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.197196 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.197434 (+ 238us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.197473 (+ 39us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.198133 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.199188 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.199706 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.199778 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.199789 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.199954 (+ 165us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.199988 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":263,"thread_start_us":58,"threads_started":1}
I20260923 23:10:27.200062 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:45015 (local address 127.0.0.1:50716)
0923 23:10:27.196758 (+ 93us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.196768 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.196829 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.197304 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.197306 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.197309 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.197368 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.197370 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.198202 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.198203 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.199118 (+ 915us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.199121 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.199788 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.199789 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.199854 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.199859 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.199940 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.199991 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82,"thread_start_us":35,"threads_started":1}
I20260923 23:10:27.200277 23772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "684dee5eabbe4479ac6605d67b0d5d98" is_pre_election: true
I20260923 23:10:27.200403 23772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12d3e8c418f245038c2c7bfe5dd3d534 in term 1.
I20260923 23:10:27.200551 23558 leader_election.cc:302] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12d3e8c418f245038c2c7bfe5dd3d534, 684dee5eabbe4479ac6605d67b0d5d98; no voters: a3802b5cce894104b40cb8e29fc478fc
I20260923 23:10:27.200657 24280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:46053 (local address 127.0.0.1:54616)
0923 23:10:27.197101 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.197111 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.197159 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.197826 (+ 667us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.197827 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.197829 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.197887 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.197889 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.198658 (+ 769us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.198659 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.199665 (+ 1006us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.199668 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.200376 (+ 708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.200378 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.200467 (+ 89us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.200472 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.200539 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.200582 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":188,"thread_start_us":33,"threads_started":1}
I20260923 23:10:27.200668 24276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 23:10:27.200708 24276 raft_consensus.cc:482] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Starting leader election (detected failure of leader a3802b5cce894104b40cb8e29fc478fc)
I20260923 23:10:27.200716 24283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.196997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54616 (local address 127.16.122.189:46053)
0923 23:10:27.197638 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.197640 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.197647 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:27.197679 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.197680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.197688 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.197788 (+ 100us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.197961 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.198583 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.199769 (+ 1186us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.200313 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.200379 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.200388 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.200593 (+ 205us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.200635 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":629,"thread_start_us":33,"threads_started":1}
I20260923 23:10:27.200732 24276 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:10:27.200888 23583 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "684dee5eabbe4479ac6605d67b0d5d98" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" is_pre_election: true
I20260923 23:10:27.201457 24276 raft_consensus.cc:504] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } }
I20260923 23:10:27.201574 24276 leader_election.cc:288] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 election: Requested vote from peers a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045), 684dee5eabbe4479ac6605d67b0d5d98 (127.16.122.188:45015)
I20260923 23:10:27.201599 23583 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 684dee5eabbe4479ac6605d67b0d5d98 in current term 2: Already voted for candidate 12d3e8c418f245038c2c7bfe5dd3d534 in this term.
I20260923 23:10:27.201769 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.201680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54218)
0923 23:10:27.201723 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.201749 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 23:10:27.201825 23743 leader_election.cc:302] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 684dee5eabbe4479ac6605d67b0d5d98; no voters: 12d3e8c418f245038c2c7bfe5dd3d534, a3802b5cce894104b40cb8e29fc478fc
I20260923 23:10:27.201849 23772 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "684dee5eabbe4479ac6605d67b0d5d98"
I20260923 23:10:27.201915 23772 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:10:27.202162 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.202047 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54220)
0923 23:10:27.202114 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.202140 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260923 23:10:27.202222 23558 leader_election.cc:334] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045): Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
I20260923 23:10:27.202579 23772 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12d3e8c418f245038c2c7bfe5dd3d534 in term 2.
I20260923 23:10:27.202642 24277 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:10:27.202739 23558 leader_election.cc:302] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 12d3e8c418f245038c2c7bfe5dd3d534, 684dee5eabbe4479ac6605d67b0d5d98; no voters: a3802b5cce894104b40cb8e29fc478fc
I20260923 23:10:27.202822 24276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:10:27.202951 24276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [term 2 LEADER]: Becoming Leader. State: Replica: 12d3e8c418f245038c2c7bfe5dd3d534, State: Running, Role: LEADER
I20260923 23:10:27.203023 24276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } }
I20260923 23:10:27.203361 24286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d3e8c418f245038c2c7bfe5dd3d534. Latest consensus state: current_term: 2 leader_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:27.203449 24286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:27.203641 24288 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:27.203970 24288 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:27.204021 24288 catalog_manager.cc:1259] Loaded cluster ID: e497fe61c14b41858bb5d98af5ca70d5
I20260923 23:10:27.204042 24288 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:27.204177 24288 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:27.204236 24288 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Loaded TSK: 0
I20260923 23:10:27.204355 24288 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:27.250559 23573 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:27.252297 23573 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:27.253365 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.253251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54238)
0923 23:10:27.253295 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.253332 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:10:27.253453 23772 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [term 2 FOLLOWER]: Refusing update from remote peer 12d3e8c418f245038c2c7bfe5dd3d534: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260923 23:10:27.253697 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.253620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.0.0.1:54240)
0923 23:10:27.253656 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.253679 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:10:27.253736 24276 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [LEADER]: Connected to new peer: Peer: permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260923 23:10:27.253855 23558 consensus_peers.cc:487] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 -> Peer a3802b5cce894104b40cb8e29fc478fc (127.16.122.190:39045): Couldn't send request to peer a3802b5cce894104b40cb8e29fc478fc. Status: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260923 23:10:27.254284 24277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 12d3e8c418f245038c2c7bfe5dd3d534. Latest consensus state: current_term: 2 leader_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:27.254372 24277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:10:27.254712 24289 mvcc.cc:205] Tried to move back new op lower bound from 7332679791627657216 to 7332679791424229376. Current Snapshot: MvccSnapshot[applied={T|T < 7332679791627657216}]
I20260923 23:10:27.254815 24276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:27.254899 24276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:27.254963 24277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" committed_config { opid_index: 9 OBSOLETE_local: true peers { permanent_uuid: "a3802b5cce894104b40cb8e29fc478fc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39045 } } peers { permanent_uuid: "12d3e8c418f245038c2c7bfe5dd3d534" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 46053 } attrs { promote: false } } peers { permanent_uuid: "684dee5eabbe4479ac6605d67b0d5d98" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 45015 } attrs { promote: false } } }
I20260923 23:10:27.255064 24277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 684dee5eabbe4479ac6605d67b0d5d98 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:10:27.255167 24290 mvcc.cc:205] Tried to move back new op lower bound from 7332679791627657216 to 7332679791424229376. Current Snapshot: MvccSnapshot[applied={T|T < 7332679791627657216 or (T in {7332679791627657216})}]
I20260923 23:10:27.259976 24278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.256790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:44843 (local address 127.0.0.1:38344)
0923 23:10:27.256838 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.256848 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.256886 (+ 38us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.257129 (+ 243us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.257130 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.257134 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.257196 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.257197 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258143 (+ 946us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.258144 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.258927 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.258929 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.259728 (+ 799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.259729 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.259791 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.259796 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.259849 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.259892 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30,"mutex_wait_us":30}
I20260923 23:10:27.260013 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.256877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38344 (local address 127.16.122.130:44843)
0923 23:10:27.256986 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.256987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.256996 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:27.257020 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.257021 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.257031 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.257257 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.257297 (+ 40us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258061 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.259014 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.259655 (+ 641us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.259725 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.259733 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.259858 (+ 125us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.259896 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":57,"threads_started":1}
I20260923 23:10:27.260227 24009 tablet_service.cc:1467] Processing CreateTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 (DEFAULT_TABLE table=test-table [id=7fc93f3b67454a499261900509afbcca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:27.260319 24279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.256931 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41871 (local address 127.0.0.1:48508)
0923 23:10:27.257086 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.257096 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.257142 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.257603 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.257603 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.257607 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.257656 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.257658 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258336 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.258337 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.259412 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.259413 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.260102 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.260103 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.260169 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.260174 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.260200 (+ 26us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.260239 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"mutex_wait_us":1,"thread_start_us":77,"threads_started":1}
I20260923 23:10:27.260360 24009 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91ec8450e5b48ac9e57034e4ae9eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:27.260442 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.256866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39943 (local address 127.0.0.1:55236)
0923 23:10:27.257063 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.257072 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.257114 (+ 42us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.257353 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.257353 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.257356 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.257388 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.257389 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258209 (+ 820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.258210 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.259388 (+ 1178us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.259390 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.260197 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.260198 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.260256 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.260260 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.260287 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.260329 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260923 23:10:27.260675 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.257124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55236 (local address 127.16.122.131:39943)
0923 23:10:27.257234 (+ 110us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.257235 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.257242 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:27.257264 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.257265 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.257275 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.257441 (+ 166us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.257475 (+ 34us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258154 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.259463 (+ 1309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.260032 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.260546 (+ 514us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.260556 (+ 10us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.260575 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.260615 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89,"thread_start_us":63,"threads_started":1}
I20260923 23:10:27.260694 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.257251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48508 (local address 127.16.122.129:41871)
0923 23:10:27.257384 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.257386 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.257393 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:27.257416 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.257417 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.257426 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.257710 (+ 284us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.257741 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.258265 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.259494 (+ 1229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.260014 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.260547 (+ 533us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.260559 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.260578 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.260620 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115,"thread_start_us":44,"threads_started":1}
I20260923 23:10:27.260885 23872 tablet_service.cc:1467] Processing CreateTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 (DEFAULT_TABLE table=test-table [id=7fc93f3b67454a499261900509afbcca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:27.261013 23872 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91ec8450e5b48ac9e57034e4ae9eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:27.261782 24299 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: Bootstrap starting.
I20260923 23:10:27.261977 24145 tablet_service.cc:1467] Processing CreateTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 (DEFAULT_TABLE table=test-table [id=7fc93f3b67454a499261900509afbcca]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:27.262089 24145 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e91ec8450e5b48ac9e57034e4ae9eb70. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:27.262194 24299 tablet_bootstrap.cc:654] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:27.262413 24300 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: Bootstrap starting.
I20260923 23:10:27.262749 24300 tablet_bootstrap.cc:654] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:27.262759 24299 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: No bootstrap required, opened a new log
I20260923 23:10:27.262801 24299 ts_tablet_manager.cc:1397] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:27.262946 24299 raft_consensus.cc:348] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.263015 24299 raft_consensus.cc:374] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:27.263075 24299 raft_consensus.cc:729] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aaa84a1d27d04e4596a3768e2953b51b, State: Initialized, Role: FOLLOWER
I20260923 23:10:27.263123 24299 consensus_queue.cc:260] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.263269 24299 ts_tablet_manager.cc:1428] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:27.263343 24304 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Bootstrap starting.
I20260923 23:10:27.263464 24300 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: No bootstrap required, opened a new log
I20260923 23:10:27.263609 24300 ts_tablet_manager.cc:1397] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:27.263684 24304 tablet_bootstrap.cc:654] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:27.263756 24300 raft_consensus.cc:348] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.263820 24300 raft_consensus.cc:374] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:27.263839 24300 raft_consensus.cc:729] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6912c866b06549f2b43547471502e863, State: Initialized, Role: FOLLOWER
I20260923 23:10:27.263875 24300 consensus_queue.cc:260] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.264012 24300 ts_tablet_manager.cc:1428] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:27.264114 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.263896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:35837)
0923 23:10:27.264016 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.264083 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":76,"threads_started":1}
I20260923 23:10:27.264739 23940 heartbeater.cc:503] Master 127.16.122.189:46053 was elected leader, sending a full tablet report...
I20260923 23:10:27.264788 24304 tablet_bootstrap.cc:492] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: No bootstrap required, opened a new log
I20260923 23:10:27.264748 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.264351 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:60305)
0923 23:10:27.264681 (+ 330us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.264716 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":310,"thread_start_us":68,"threads_started":1}
I20260923 23:10:27.264827 24304 ts_tablet_manager.cc:1397] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:27.264925 24304 raft_consensus.cc:348] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.264989 24304 raft_consensus.cc:374] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:27.265007 24304 raft_consensus.cc:729] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c89e3f7e62424ca7ade97bcdebf4bb93, State: Initialized, Role: FOLLOWER
I20260923 23:10:27.265062 24304 consensus_queue.cc:260] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.265139 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.265059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:59249)
0923 23:10:27.265093 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.265115 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":20}
I20260923 23:10:27.265183 24304 ts_tablet_manager.cc:1428] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:27.265450 24077 heartbeater.cc:503] Master 127.16.122.189:46053 was elected leader, sending a full tablet report...
I20260923 23:10:27.265646 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.265443 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:36727)
0923 23:10:27.265578 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.265614 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":74,"threads_started":1}
I20260923 23:10:27.265733 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.265387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:55339)
0923 23:10:27.265675 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.265706 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 23:10:27.265781 24213 heartbeater.cc:503] Master 127.16.122.189:46053 was elected leader, sending a full tablet report...
I20260923 23:10:27.266003 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.265922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:54569)
0923 23:10:27.265953 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.265981 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260923 23:10:27.285028 23941 tablet.cc:2369] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:27.318645 24214 tablet.cc:2369] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:27.347895 24303 raft_consensus.cc:482] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:27.347995 24303 raft_consensus.cc:504] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.348307 24303 leader_election.cc:288] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943), 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:27.352407 24297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.348666 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:45397 (local address 127.16.122.131:39943)
0923 23:10:27.348727 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.348728 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.348787 (+ 59us) server_negotiation.cc:407] Connection header received
0923 23:10:27.348858 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.348859 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.348869 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.348933 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.349113 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.349871 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.351417 (+ 1546us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.351995 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.352079 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.352091 (+ 12us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.352274 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.352319 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:10:27.352407 24298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.348826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:57923 (local address 127.16.122.129:41871)
0923 23:10:27.348867 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.348869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.348968 (+ 99us) server_negotiation.cc:407] Connection header received
0923 23:10:27.349027 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.349028 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.349034 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.349073 (+ 39us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:27.349185 (+ 112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.349877 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.351417 (+ 1540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.351993 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.352079 (+ 86us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.352112 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.352273 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.352318 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:27.352443 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.348602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39943 (local address 127.16.122.130:45397)
0923 23:10:27.348699 (+ 97us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.348711 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.348793 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.348965 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.348966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.348969 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.349044 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.349047 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.349969 (+ 922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.349970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.351304 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.351308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.352140 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.352140 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.352191 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.352196 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.352257 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.352310 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":74,"mutex_wait_us":44}
I20260923 23:10:27.352464 24311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.348721 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41871 (local address 127.16.122.130:57923)
0923 23:10:27.348901 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.348909 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:27.348965 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:27.349092 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:27.349093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:27.349094 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:27.349125 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.349127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.349969 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.349970 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.351304 (+ 1334us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:27.351308 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.352117 (+ 809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.352118 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:27.352191 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.352196 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:27.352257 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:10:27.352310 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":163,"mutex_wait_us":59,"thread_start_us":119,"threads_started":1}
I20260923 23:10:27.352838 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91ec8450e5b48ac9e57034e4ae9eb70" candidate_uuid: "aaa84a1d27d04e4596a3768e2953b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6912c866b06549f2b43547471502e863" is_pre_election: true
I20260923 23:10:27.352833 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91ec8450e5b48ac9e57034e4ae9eb70" candidate_uuid: "aaa84a1d27d04e4596a3768e2953b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" is_pre_election: true
I20260923 23:10:27.352926 23892 raft_consensus.cc:2454] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa84a1d27d04e4596a3768e2953b51b in term 0.
I20260923 23:10:27.352941 24165 raft_consensus.cc:2454] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate aaa84a1d27d04e4596a3768e2953b51b in term 0.
I20260923 23:10:27.353101 23963 leader_election.cc:302] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa84a1d27d04e4596a3768e2953b51b, c89e3f7e62424ca7ade97bcdebf4bb93; no voters:
I20260923 23:10:27.353304 24303 raft_consensus.cc:2781] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:27.353338 24303 raft_consensus.cc:482] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:27.353350 24303 raft_consensus.cc:3037] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:27.354012 24303 raft_consensus.cc:504] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.354123 24303 leader_election.cc:288] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [CANDIDATE]: Term 1 election: Requested vote from peers c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943), 6912c866b06549f2b43547471502e863 (127.16.122.129:41871)
I20260923 23:10:27.354288 24165 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91ec8450e5b48ac9e57034e4ae9eb70" candidate_uuid: "aaa84a1d27d04e4596a3768e2953b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93"
I20260923 23:10:27.354353 24165 raft_consensus.cc:3037] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:27.354347 23892 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e91ec8450e5b48ac9e57034e4ae9eb70" candidate_uuid: "aaa84a1d27d04e4596a3768e2953b51b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6912c866b06549f2b43547471502e863"
I20260923 23:10:27.354430 23892 raft_consensus.cc:3037] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:27.354987 24165 raft_consensus.cc:2454] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa84a1d27d04e4596a3768e2953b51b in term 1.
I20260923 23:10:27.355090 23892 raft_consensus.cc:2454] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate aaa84a1d27d04e4596a3768e2953b51b in term 1.
I20260923 23:10:27.355157 23963 leader_election.cc:302] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aaa84a1d27d04e4596a3768e2953b51b, c89e3f7e62424ca7ade97bcdebf4bb93; no voters:
I20260923 23:10:27.355248 24303 raft_consensus.cc:2781] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:27.355403 24303 raft_consensus.cc:686] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 1 LEADER]: Becoming Leader. State: Replica: aaa84a1d27d04e4596a3768e2953b51b, State: Running, Role: LEADER
I20260923 23:10:27.355490 24303 consensus_queue.cc:237] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } }
I20260923 23:10:27.355733 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.355628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:41369)
0923 23:10:27.355672 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.355702 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:10:27.356215 23572 catalog_manager.cc:5495] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b reported cstate change: term changed from 0 to 1, leader changed from <none> to aaa84a1d27d04e4596a3768e2953b51b (127.16.122.130). New cstate: current_term: 1 leader_uuid: "aaa84a1d27d04e4596a3768e2953b51b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aaa84a1d27d04e4596a3768e2953b51b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 44843 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:27.377141 24295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.374985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38358 (local address 127.16.122.130:44843)
0923 23:10:27.375064 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:27.375066 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:27.375073 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:27.375109 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:27.375110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:27.375129 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:27.375190 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:27.375301 (+ 111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.375764 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.376460 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:27.376651 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:27.376705 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:27.376836 (+ 131us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:27.376934 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:27.376993 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:27.377039 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 23:10:27.377060 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:10:27.378197 23892 raft_consensus.cc:1261] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Refusing update from remote peer aaa84a1d27d04e4596a3768e2953b51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:27.378326 24165 raft_consensus.cc:1261] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Refusing update from remote peer aaa84a1d27d04e4596a3768e2953b51b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:27.378451 24303 consensus_queue.cc:1035] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "6912c866b06549f2b43547471502e863" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:27.378708 24303 consensus_queue.cc:1035] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [LEADER]: Connected to new peer: Peer: permanent_uuid: "c89e3f7e62424ca7ade97bcdebf4bb93" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39943 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:27.379158 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.379020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:50037)
0923 23:10:27.379084 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.379127 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 23:10:27.379635 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.379485 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:44745)
0923 23:10:27.379565 (+ 80us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.379603 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20260923 23:10:27.380947 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.380792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:43005)
0923 23:10:27.380890 (+ 98us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.380917 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 23:10:27.381305 24310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.381203 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.131:34105)
0923 23:10:27.381258 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.381280 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 23:10:27.381405 24308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.381292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.129:56995)
0923 23:10:27.381355 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.381378 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
I20260923 23:10:27.381626 24306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.381549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39045 (local address 127.16.122.130:58581)
0923 23:10:27.381582 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.381602 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:39045: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
I20260923 23:10:27.381721 24318 mvcc.cc:205] Tried to move back new op lower bound from 7332679792139141120 to 7332679792048570368. Current Snapshot: MvccSnapshot[applied={T|T < 7332679792139141120 or (T in {7332679792139141120})}]
I20260923 23:10:27.383701 23573 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:27.383790 23573 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:54608:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:27.384951 23573 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 12d3e8c418f245038c2c7bfe5dd3d534: Sending DeleteTablet for 3 replicas of tablet e91ec8450e5b48ac9e57034e4ae9eb70
I20260923 23:10:27.385262 24145 tablet_service.cc:1514] Processing DeleteTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:27 UTC) from {username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:55236
I20260923 23:10:27.385329 23872 tablet_service.cc:1514] Processing DeleteTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:27 UTC) from {username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:48508
I20260923 23:10:27.385499 24326 tablet_replica.cc:330] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863: stopping tablet replica
I20260923 23:10:27.385589 24326 raft_consensus.cc:2229] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:27.385578 24009 tablet_service.cc:1514] Processing DeleteTablet for tablet e91ec8450e5b48ac9e57034e4ae9eb70 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:27 UTC) from {username='oryx', principal='oryx/127.16.122.189@KRBTEST.COM'} at 127.0.0.1:38344
I20260923 23:10:27.385648 24326 raft_consensus.cc:2258] T e91ec8450e5b48ac9e57034e4ae9eb70 P 6912c866b06549f2b43547471502e863 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:27.385936 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23721
I20260923 23:10:27.387283 24327 tablet_replica.cc:330] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: stopping tablet replica
I20260923 23:10:27.387348 24327 raft_consensus.cc:2229] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:27.387439 24327 raft_consensus.cc:2258] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:27.387530 24325 tablet_replica.cc:330] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: stopping tablet replica
I20260923 23:10:27.387571 24325 raft_consensus.cc:2229] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:27.387607 24325 raft_consensus.cc:2258] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:27.387885 24325 ts_tablet_manager.cc:1905] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:27.389524 24325 ts_tablet_manager.cc:1918] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:27.389590 24325 log.cc:1196] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/e91ec8450e5b48ac9e57034e4ae9eb70
I20260923 23:10:27.389818 24325 ts_tablet_manager.cc:1939] T e91ec8450e5b48ac9e57034e4ae9eb70 P c89e3f7e62424ca7ade97bcdebf4bb93: Deleting consensus metadata
I20260923 23:10:27.389930 24327 ts_tablet_manager.cc:1905] T e91ec8450e5b48ac9e57034e4ae9eb70 P aaa84a1d27d04e4596a3768e2953b51b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:27.390261 23558 catalog_manager.cc:4841] TS c89e3f7e62424ca7ade97bcdebf4bb93 (127.16.122.131:39943): tablet e91ec8450e5b48ac9e57034e4ae9eb70 (table test-table [id=7fc93f3b67454a499261900509afbcca]) successfully deleted
I20260923 23:10:27.390918 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23946
I20260923 23:10:27.390987 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.390886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41871 (local address 127.0.0.1:48514)
0923 23:10:27.390928 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.390952 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:41871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 23:10:27.391683 23558 catalog_manager.cc:4584] TS 6912c866b06549f2b43547471502e863 (127.16.122.129:41871): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e91ec8450e5b48ac9e57034e4ae9eb70: Network error: Client connection negotiation failed: client connection to 127.16.122.129:41871: connect: Connection refused (error 111)
I20260923 23:10:27.395354 24296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:27.395248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:44843 (local address 127.0.0.1:38360)
0923 23:10:27.395299 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:27.395324 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.130:44843: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 23:10:27.395443 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 24082
I20260923 23:10:27.400520 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23437
I20260923 23:10:27.404871 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 23627
2026-09-23T23:10:27Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (4419 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:27 dist-test-slave-95hh krb5kdc[24330](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:27 dist-test-slave-95hh krb5kdc[24330](info): set up 2 sockets
Sep 23 23:10:27 dist-test-slave-95hh krb5kdc[24330](info): commencing operation
krb5kdc: starting...
W20260923 23:10:28.363823 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:28Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:28Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab.
Entry for principal oryx/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab.
I20260923 23:10:28.474982 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37961
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab
--principal=oryx/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:37961
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:28.554569 24343 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:28.554781 24343 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:28.554801 24343 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:28.555969 24343 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:28.556016 24343 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260923 23:10:28.556030 24343 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:28.556051 24343 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:28.556073 24343 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:28.556094 24343 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:28.557691 24343 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:37961
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.190.keytab
--principal=oryx/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37961
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24343
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:28.557950 24343 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:28.558175 24343 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:28.563004 24348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:28.563056 24351 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.563050 24343 server_base.cc:1034] running on GCE node
W20260923 23:10:28.563002 24349 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.563448 24343 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:28.563722 24343 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:28.564857 24343 hybrid_clock.cc:648] HybridClock initialized: now 1790205028564841 us; error 35 us; skew 500 ppm
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:28.567155 24343 init.cc:377] Logged in from keytab as oryx/127.16.122.190@KRBTEST.COM (short username oryx)
I20260923 23:10:28.567559 24343 webserver.cc:460] Webserver started at http://127.16.122.190:42445/ using document root <none> and password file <none>
I20260923 23:10:28.567745 24343 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:28.567783 24343 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:28.567873 24343 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:28.568846 24343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "f9355750eaf64779adc1a6ad15c718a7"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "2e29776a9583b27ff6266b82f4467e3c"
server_key_iv: "3df13ac409c1c23b134d5f78cb6ead40"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.569128 24343 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "f9355750eaf64779adc1a6ad15c718a7"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "2e29776a9583b27ff6266b82f4467e3c"
server_key_iv: "3df13ac409c1c23b134d5f78cb6ead40"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.570287 24343 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:28.570901 24358 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.571069 24343 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:28.571146 24343 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "f9355750eaf64779adc1a6ad15c718a7"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "2e29776a9583b27ff6266b82f4467e3c"
server_key_iv: "3df13ac409c1c23b134d5f78cb6ead40"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.571199 24343 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:28.586454 24343 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:28.588196 24343 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:28.588302 24343 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:28.592463 24343 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:37961
I20260923 23:10:28.592528 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:37961 every 8 connection(s)
I20260923 23:10:28.592804 24343 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:28.593427 24421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:28.595551 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Bootstrap starting.
I20260923 23:10:28.596091 24421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:28.596344 24421 log.cc:824] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:28.596968 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: No bootstrap required, opened a new log
I20260923 23:10:28.598996 24421 raft_consensus.cc:348] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } }
I20260923 23:10:28.599164 24421 raft_consensus.cc:374] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:28.599197 24421 raft_consensus.cc:729] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f9355750eaf64779adc1a6ad15c718a7, State: Initialized, Role: FOLLOWER
I20260923 23:10:28.599334 24421 consensus_queue.cc:260] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } }
I20260923 23:10:28.599407 24421 raft_consensus.cc:388] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:28.599437 24421 raft_consensus.cc:482] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:28.599488 24421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:28.599607 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 24343
I20260923 23:10:28.599694 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:28.599877 16874 external_mini_cluster.cc:1428] Setting key 04035d40bfa99855dc0c41a8de6c5416
I20260923 23:10:28.600137 24421 raft_consensus.cc:504] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } }
I20260923 23:10:28.600235 24421 leader_election.cc:302] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f9355750eaf64779adc1a6ad15c718a7; no voters:
I20260923 23:10:28.600397 24421 leader_election.cc:288] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:28.600499 24425 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:28.600634 24425 raft_consensus.cc:686] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [term 1 LEADER]: Becoming Leader. State: Replica: f9355750eaf64779adc1a6ad15c718a7, State: Running, Role: LEADER
I20260923 23:10:28.600646 24421 sys_catalog.cc:564] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:28.600733 24425 consensus_queue.cc:237] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } }
I20260923 23:10:28.601043 24425 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f9355750eaf64779adc1a6ad15c718a7. Latest consensus state: current_term: 1 leader_uuid: "f9355750eaf64779adc1a6ad15c718a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } } }
I20260923 23:10:28.601123 24425 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:28.601052 24428 sys_catalog.cc:455] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f9355750eaf64779adc1a6ad15c718a7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f9355750eaf64779adc1a6ad15c718a7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37961 } } }
I20260923 23:10:28.601274 24428 sys_catalog.cc:458] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7 [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:28.602362 24441 catalog_manager.cc:1558] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:28.602418 24441 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:28.602509 24433 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:28.602684 24433 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:28.609185 24433 catalog_manager.cc:1347] Generated new cluster ID: f6ed55bc64844988a6cdf9d263241b3f
I20260923 23:10:28.609241 24433 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:28.611555 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.600343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39100 (local address 127.16.122.190:37961)
0923 23:10:28.600499 (+ 156us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:28.600510 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:28.600684 (+ 174us) server_negotiation.cc:407] Connection header received
0923 23:10:28.600808 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:28.600815 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:28.600902 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:28.601034 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:28.601961 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.603548 (+ 1587us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.604621 (+ 1073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.605270 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.605387 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.608139 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:28.608143 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:28.608143 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:28.608156 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:28.609949 (+ 1793us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.610343 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.610345 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.610346 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.610443 (+ 97us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.610637 (+ 194us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.610638 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.610638 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.610917 (+ 279us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:28.611091 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:28.611210 (+ 119us) server_negotiation.cc:299] Negotiation successful
0923 23:10:28.611342 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":69,"threads_started":1}
I20260923 23:10:28.614463 24433 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:28.615206 24433 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:28.620258 24433 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Generated new TSK 0
I20260923 23:10:28.620435 24433 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
I20260923 23:10:28.651276 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37961
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:28.756014 24447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:28.756218 24447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:28.756242 24447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:28.757336 24447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:28.757373 24447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:28.757392 24447 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:28.757408 24447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:28.758810 24447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24447
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:28.758986 24447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:28.759209 24447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:28.763985 24453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.764020 24447 server_base.cc:1034] running on GCE node
W20260923 23:10:28.763943 24452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:28.763943 24455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.764317 24447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:28.764554 24447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:28.765692 24447 hybrid_clock.cc:648] HybridClock initialized: now 1790205028765676 us; error 29 us; skew 500 ppm
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:28.768246 24447 init.cc:377] Logged in from keytab as oryx/127.16.122.129@KRBTEST.COM (short username oryx)
I20260923 23:10:28.768654 24447 webserver.cc:460] Webserver started at http://127.16.122.129:46133/ using document root <none> and password file <none>
I20260923 23:10:28.768849 24447 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:28.768896 24447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:28.769017 24447 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:28.770026 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "a94a7a36a360438bb73e8d5e34b63b03"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "abbce64bc6cf0902ec57fbc653559535"
server_key_iv: "83e53a415d873421b9df847358123b22"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.770349 24447 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "a94a7a36a360438bb73e8d5e34b63b03"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "abbce64bc6cf0902ec57fbc653559535"
server_key_iv: "83e53a415d873421b9df847358123b22"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.771595 24447 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.001s
I20260923 23:10:28.772255 24462 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.772408 24447 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.772487 24447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "a94a7a36a360438bb73e8d5e34b63b03"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "abbce64bc6cf0902ec57fbc653559535"
server_key_iv: "83e53a415d873421b9df847358123b22"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.772543 24447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:28.799633 24447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:28.801892 24447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:28.802026 24447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:28.802541 24447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:28.802582 24447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.802604 24447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:28.802618 24447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:28.809291 24447 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:34125
I20260923 23:10:28.809381 24577 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:34125 every 8 connection(s)
I20260923 23:10:28.809621 24447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:28.811568 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.803320 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:37323 (local address 127.16.122.190:37961)
0923 23:10:28.803382 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:28.803384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:28.803395 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:28.803862 (+ 467us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:28.803863 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:28.803879 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:28.803945 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:28.804853 (+ 908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.805398 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.806379 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.806633 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.806687 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.809115 (+ 2428us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:28.809117 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:28.809117 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:28.809127 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:28.810511 (+ 1384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.810832 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.810833 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.810833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.810878 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.811054 (+ 176us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.811055 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.811055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.811162 (+ 107us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:28.811223 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:28.811440 (+ 217us) server_negotiation.cc:299] Negotiation successful
0923 23:10:28.811505 (+ 65us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:10:28.811719 24471 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.803069 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.129:37323)
0923 23:10:28.803204 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:28.803238 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:28.803716 (+ 478us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:28.804080 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:28.804085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:28.804432 (+ 347us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:28.804730 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.804738 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.805483 (+ 745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.805484 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.806289 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.806291 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.806720 (+ 429us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.806721 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.806852 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.807344 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:28.807356 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:28.809029 (+ 1673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:28.810577 (+ 1548us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.810581 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.810583 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.810777 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.810912 (+ 135us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.810913 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.810913 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.810991 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.811222 (+ 231us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:28.811225 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:28.811335 (+ 110us) client_negotiation.cc:769] Sending connection context
0923 23:10:28.811423 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:10:28.811543 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":75,"threads_started":1}
I20260923 23:10:28.812141 24578 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37961
I20260923 23:10:28.812213 24578 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:28.812371 24578 heartbeater.cc:511] Master 127.16.122.190:37961 requested a full tablet report, sending...
I20260923 23:10:28.812907 24374 ts_manager.cc:194] Registered new tserver with Master: a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125)
I20260923 23:10:28.813582 24374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:37323
I20260923 23:10:28.816645 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 24447
I20260923 23:10:28.816736 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:28.816912 16874 external_mini_cluster.cc:1428] Setting key 8196cc61ece52328c67dd1ec797fbf1f
I20260923 23:10:28.818843 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.813738 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:57519 (local address 127.16.122.190:37961)
0923 23:10:28.813784 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:28.813785 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:28.813793 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:28.813807 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:28.813808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:28.813815 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:28.813862 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:28.814157 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.814607 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.815274 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.815451 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.815528 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.816233 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:28.816235 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:28.816235 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:28.816241 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:28.817965 (+ 1724us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.818241 (+ 276us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.818242 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.818242 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.818294 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.818432 (+ 138us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.818432 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.818432 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.818545 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:28.818600 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:28.818697 (+ 97us) server_negotiation.cc:299] Negotiation successful
0923 23:10:28.818765 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:10:28.818851 24584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.813032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.129:57519)
0923 23:10:28.813139 (+ 107us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:28.813150 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:28.813200 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:28.813876 (+ 676us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:28.813877 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:28.814037 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:28.814096 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.814098 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.814683 (+ 585us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.814684 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.815215 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.815217 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.815504 (+ 287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.815505 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.815566 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.815909 (+ 343us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:28.815913 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:28.816161 (+ 248us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:28.818033 (+ 1872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.818034 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.818034 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.818181 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.818342 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.818343 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.818343 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.818386 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.818592 (+ 206us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:28.818593 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:28.818623 (+ 30us) client_negotiation.cc:769] Sending connection context
0923 23:10:28.818676 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:28.818755 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"thread_start_us":58,"threads_started":1}
WARNING: no policy specified for oryx/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
I20260923 23:10:28.839614 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37961
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:28.917811 24587 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:28.918018 24587 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:28.918051 24587 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:28.919240 24587 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:28.919279 24587 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:28.919307 24587 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:28.919332 24587 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:28.920857 24587 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24587
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:28.921093 24587 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:28.921314 24587 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:28.926223 24593 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.926290 24587 server_base.cc:1034] running on GCE node
W20260923 23:10:28.926244 24595 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:28.926244 24592 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:28.926611 24587 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:28.926837 24587 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:28.927966 24587 hybrid_clock.cc:648] HybridClock initialized: now 1790205028927939 us; error 42 us; skew 500 ppm
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:28.930287 24587 init.cc:377] Logged in from keytab as oryx/127.16.122.130@KRBTEST.COM (short username oryx)
I20260923 23:10:28.930699 24587 webserver.cc:460] Webserver started at http://127.16.122.130:36651/ using document root <none> and password file <none>
I20260923 23:10:28.930905 24587 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:28.930946 24587 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:28.931094 24587 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:28.932108 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "ff40c6dad56246ff83aac93b0904eba1"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "34f78a9e1efac6b59a193fc21c121d9a"
server_key_iv: "c91fb4a03c6f8068745de68b2b287484"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.932411 24587 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "ff40c6dad56246ff83aac93b0904eba1"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "34f78a9e1efac6b59a193fc21c121d9a"
server_key_iv: "c91fb4a03c6f8068745de68b2b287484"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.933564 24587 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:28.934219 24602 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.934401 24587 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:28.934480 24587 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "ff40c6dad56246ff83aac93b0904eba1"
format_stamp: "Formatted at 2026-09-23 23:10:28 on dist-test-slave-95hh"
server_key: "34f78a9e1efac6b59a193fc21c121d9a"
server_key_iv: "c91fb4a03c6f8068745de68b2b287484"
server_key_version: "encryptionkey@0"
I20260923 23:10:28.934540 24587 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:28.944226 24587 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:28.945819 24587 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:28.945940 24587 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:28.946436 24587 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:28.946473 24587 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:28.946501 24587 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:28.946524 24587 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:28 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:28.952971 24587 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:46113
I20260923 23:10:28.953038 24717 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:46113 every 8 connection(s)
I20260923 23:10:28.953317 24587 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:28.953660 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 24587
I20260923 23:10:28.953780 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:28.953969 16874 external_mini_cluster.cc:1428] Setting key 1edda0b434d0ec9fb03315e8363837b0
I20260923 23:10:28.955910 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.946765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:60989 (local address 127.16.122.190:37961)
0923 23:10:28.946825 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:28.946827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:28.946984 (+ 157us) server_negotiation.cc:407] Connection header received
0923 23:10:28.947502 (+ 518us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:28.947503 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:28.947515 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:28.947591 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:28.948158 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.948892 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.949861 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.950142 (+ 281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.950223 (+ 81us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.952627 (+ 2404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:28.952629 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:28.952629 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:28.952637 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:28.954616 (+ 1979us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.954966 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.954967 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.954968 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.955052 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.955317 (+ 265us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.955318 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.955318 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.955416 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:28.955461 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:28.955744 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 23:10:28.955817 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:28.956108 24611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.946733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.130:60989)
0923 23:10:28.946862 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:28.946900 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:28.947412 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:28.947642 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:28.947645 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:28.947870 (+ 225us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:28.948088 (+ 218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.948093 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.948986 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.948987 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.949797 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.949800 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.950256 (+ 456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.950257 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.950388 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.950843 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:28.950850 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:28.952538 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:28.954687 (+ 2149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.954693 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.954695 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.954901 (+ 206us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.955110 (+ 209us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.955111 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.955112 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.955258 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.955467 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:28.955471 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:28.955620 (+ 149us) client_negotiation.cc:769] Sending connection context
0923 23:10:28.955720 (+ 100us) client_negotiation.cc:240] Negotiation successful
0923 23:10:28.955889 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":71,"threads_started":1}
I20260923 23:10:28.956542 24718 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37961
I20260923 23:10:28.956635 24718 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:28.956811 24718 heartbeater.cc:511] Master 127.16.122.190:37961 requested a full tablet report, sending...
I20260923 23:10:28.957208 24375 ts_manager.cc:194] Registered new tserver with Master: ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113)
I20260923 23:10:28.957674 24375 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:60989
WARNING: no policy specified for oryx/127.16.122.131@KRBTEST.COM; defaulting to no policy
I20260923 23:10:28.963346 24725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.957895 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.130:35857)
0923 23:10:28.958039 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:28.958050 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:28.958109 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:28.958216 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:28.958217 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:28.958362 (+ 145us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:28.958417 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.958418 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.959255 (+ 837us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.959256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.959771 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:28.959772 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.960191 (+ 419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.960191 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:28.960236 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.960693 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:28.960697 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:28.960948 (+ 251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:28.962425 (+ 1477us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.962426 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.962426 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.962572 (+ 146us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.962749 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:28.962750 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:28.962750 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:28.962793 (+ 43us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:28.963106 (+ 313us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:28.963107 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:28.963150 (+ 43us) client_negotiation.cc:769] Sending connection context
0923 23:10:28.963203 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:28.963274 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":63,"threads_started":1}
I20260923 23:10:28.963387 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:28.957943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:35857 (local address 127.16.122.190:37961)
0923 23:10:28.957981 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:28.957983 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:28.958098 (+ 115us) server_negotiation.cc:407] Connection header received
0923 23:10:28.958154 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:28.958155 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:28.958165 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:28.958213 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:28.958480 (+ 267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.959180 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.959840 (+ 660us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:28.960120 (+ 280us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:28.960186 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:28.961027 (+ 841us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:28.961029 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:28.961029 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:28.961036 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:28.962353 (+ 1317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.962636 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.962637 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.962637 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.962693 (+ 56us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:28.962827 (+ 134us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:28.962828 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:28.962828 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:28.963002 (+ 174us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:28.963113 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:28.963244 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:10:28.963311 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
I20260923 23:10:28.979281 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37961
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43671
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:29.059341 24727 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:29.059558 24727 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:29.059581 24727 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:29.060734 24727 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:29.060771 24727 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:29.060791 24727 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260923 23:10:29.060809 24727 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:29.062250 24727 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43671
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.122.190:37961
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24727
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:29.062431 24727 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:29.062618 24727 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:29.067310 24732 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:29.067328 24733 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:29.067312 24735 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:29.067584 24727 server_base.cc:1034] running on GCE node
I20260923 23:10:29.067782 24727 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:29.067989 24727 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:29.069139 24727 hybrid_clock.cc:648] HybridClock initialized: now 1790205029069124 us; error 28 us; skew 500 ppm
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:29.071623 24727 init.cc:377] Logged in from keytab as oryx/127.16.122.131@KRBTEST.COM (short username oryx)
I20260923 23:10:29.072084 24727 webserver.cc:460] Webserver started at http://127.16.122.131:34573/ using document root <none> and password file <none>
I20260923 23:10:29.072283 24727 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:29.072329 24727 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:29.072451 24727 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:29.073493 24727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "b1045df71067405db80c3626137cb8a5"
format_stamp: "Formatted at 2026-09-23 23:10:29 on dist-test-slave-95hh"
server_key: "d2e24da54eaed55a99e859e129a5ae72"
server_key_iv: "074e4cfad4f4bf9909597aedfa77da2c"
server_key_version: "encryptionkey@0"
I20260923 23:10:29.073827 24727 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "b1045df71067405db80c3626137cb8a5"
format_stamp: "Formatted at 2026-09-23 23:10:29 on dist-test-slave-95hh"
server_key: "d2e24da54eaed55a99e859e129a5ae72"
server_key_iv: "074e4cfad4f4bf9909597aedfa77da2c"
server_key_version: "encryptionkey@0"
I20260923 23:10:29.075225 24727 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:29.076025 24742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.076186 24727 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:29.076273 24727 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "b1045df71067405db80c3626137cb8a5"
format_stamp: "Formatted at 2026-09-23 23:10:29 on dist-test-slave-95hh"
server_key: "d2e24da54eaed55a99e859e129a5ae72"
server_key_iv: "074e4cfad4f4bf9909597aedfa77da2c"
server_key_version: "encryptionkey@0"
I20260923 23:10:29.076344 24727 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:29.086458 24727 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:29.088310 24727 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:29.088444 24727 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:29.088963 24727 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:29.089004 24727 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.089033 24727 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:29.089049 24727 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:29.095628 24727 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:40553
I20260923 23:10:29.095690 24857 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:40553 every 8 connection(s)
I20260923 23:10:29.096015 24727 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:29.098205 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.089338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:45717 (local address 127.16.122.190:37961)
0923 23:10:29.089396 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.089399 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.089642 (+ 243us) server_negotiation.cc:407] Connection header received
0923 23:10:29.090119 (+ 477us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.090120 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.090132 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.090211 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.090953 (+ 742us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.091548 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.092513 (+ 965us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.092713 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.092767 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.095774 (+ 3007us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:29.095776 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:29.095777 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:29.095785 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:29.097132 (+ 1347us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.097465 (+ 333us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.097466 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.097466 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.097511 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.097726 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.097726 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.097726 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.097821 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:29.097864 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.098078 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.098137 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:29.098387 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.089365 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.131:45717)
0923 23:10:29.089515 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.089556 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.090032 (+ 476us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.090279 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.090282 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.090567 (+ 285us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:29.090857 (+ 290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.090864 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.091623 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.091624 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.092435 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.092438 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.092777 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.092778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.092894 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.093423 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:29.093434 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:29.095692 (+ 2258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:29.097197 (+ 1505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.097202 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.097203 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.097405 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.097578 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.097579 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.097579 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.097681 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.097865 (+ 184us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:29.097867 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:29.097979 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.098060 (+ 81us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.098181 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":73,"threads_started":1}
I20260923 23:10:29.098824 24858 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37961
I20260923 23:10:29.098902 24858 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:29.099118 24858 heartbeater.cc:511] Master 127.16.122.190:37961 requested a full tablet report, sending...
I20260923 23:10:29.099612 24374 ts_manager.cc:194] Registered new tserver with Master: b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553)
I20260923 23:10:29.100068 24374 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:45717
I20260923 23:10:29.103473 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 24727
I20260923 23:10:29.103600 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:29.103812 16874 external_mini_cluster.cc:1428] Setting key f8c8678f6484ff70b3c273cb038f8458
I20260923 23:10:29.104600 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:10:29.105058 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.099823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.16.122.131:58573)
0923 23:10:29.099960 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.099969 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.100035 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.100161 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.100162 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.100309 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:29.100364 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.100365 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.101172 (+ 807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.101173 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.101737 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.101739 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.102030 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.102031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.102088 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.102463 (+ 375us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:29.102467 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:29.102730 (+ 263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:29.104145 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.104146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.104146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.104354 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.104525 (+ 171us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.104526 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.104526 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.104590 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.104827 (+ 237us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:29.104828 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:29.104864 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.104910 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.104984 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":70,"threads_started":1}
I20260923 23:10:29.105096 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.099895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:58573 (local address 127.16.122.190:37961)
0923 23:10:29.099935 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.099937 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.100008 (+ 71us) server_negotiation.cc:407] Connection header received
0923 23:10:29.100082 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.100083 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.100092 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.100143 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.100435 (+ 292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.101058 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.101796 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.101977 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.102030 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.102791 (+ 761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:29.102792 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:29.102792 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:29.102797 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:29.104063 (+ 1266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.104413 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.104414 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.104414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.104466 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.104645 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.104646 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.104646 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.104766 (+ 120us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:29.104822 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.104942 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.105009 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:10:29.124656 24426 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:29.119832 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39102 (local address 127.16.122.190:37961)
0923 23:10:29.119891 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.119893 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.119903 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:29.119929 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.119930 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.119938 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.119986 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.120521 (+ 535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.121002 (+ 481us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.121695 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.121880 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.121930 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.124578 (+ 2648us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39102: BlockingRecv error: recv got EOF from 127.0.0.1:39102 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:29.130836 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.125216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39116 (local address 127.16.122.190:37961)
0923 23:10:29.125259 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.125261 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.125321 (+ 60us) server_negotiation.cc:407] Connection header received
0923 23:10:29.125359 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.125359 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.125367 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.125413 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.125819 (+ 406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.126265 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.126876 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.127057 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.127110 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.128905 (+ 1795us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:29.128906 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:29.128906 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:29.128912 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:29.130057 (+ 1145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.130336 (+ 279us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.130337 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.130337 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.130379 (+ 42us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.130500 (+ 121us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.130500 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.130500 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.130586 (+ 86us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:29.130649 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.130719 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.130772 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:10:29.132293 24374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:29.133806 24374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:29.140018 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.136094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40028 (local address 127.16.122.129:34125)
0923 23:10:29.136268 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.136275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.136331 (+ 56us) server_negotiation.cc:407] Connection header received
0923 23:10:29.136400 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.136403 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.136418 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.136530 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.136668 (+ 138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.137798 (+ 1130us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139159 (+ 1361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.139634 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139718 (+ 84us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.139777 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.139885 (+ 108us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.139922 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":68,"threads_started":1}
I20260923 23:10:29.140005 24881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.135969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34125 (local address 127.0.0.1:40028)
0923 23:10:29.136263 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.136273 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.136329 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.136517 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.136520 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.136522 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.136592 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.136595 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.137882 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.137883 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.139081 (+ 1198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.139082 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.139715 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139716 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.139790 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.139803 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.139871 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.139922 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":280,"mutex_wait_us":1,"thread_start_us":55,"threads_started":1}
I20260923 23:10:29.140257 24880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.135966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:46113 (local address 127.0.0.1:47998)
0923 23:10:29.136124 (+ 158us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.136163 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.136225 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.136637 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.136639 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.136641 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.136693 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.136695 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.137703 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.137704 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.139052 (+ 1348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.139055 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.140039 (+ 984us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.140040 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.140100 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.140104 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.140146 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.140185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":22,"thread_start_us":91,"threads_started":1}
I20260923 23:10:29.140497 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.136067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47998 (local address 127.16.122.130:46113)
0923 23:10:29.136509 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.136513 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.136522 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:29.136557 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.136559 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.136572 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.136638 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.136849 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.137633 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139126 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.139863 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139956 (+ 93us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.140005 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.140365 (+ 360us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.140405 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":407,"thread_start_us":86,"threads_started":1}
I20260923 23:10:29.140983 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 16c759d24f8e45cd9346f16e00147356 (DEFAULT_TABLE table=test-table [id=e09344932cf045c281b39c3842c40ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.141079 24652 tablet_service.cc:1467] Processing CreateTablet for tablet 16c759d24f8e45cd9346f16e00147356 (DEFAULT_TABLE table=test-table [id=e09344932cf045c281b39c3842c40ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.141374 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c759d24f8e45cd9346f16e00147356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.141374 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c759d24f8e45cd9346f16e00147356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.141518 24884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.136158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:40553 (local address 127.0.0.1:37196)
0923 23:10:29.136325 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.136333 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.136369 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.136662 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.136663 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.136664 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.136714 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.136715 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.139132 (+ 2417us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.139133 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.140208 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.140210 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.141312 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.141313 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.141368 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.141371 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.141425 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.141459 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":150,"mutex_wait_us":13,"thread_start_us":62,"threads_started":1}
I20260923 23:10:29.141774 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.136314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37196 (local address 127.16.122.131:40553)
0923 23:10:29.136464 (+ 150us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.136468 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.136476 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:29.136513 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.136515 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.136531 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.136604 (+ 73us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.136783 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.137807 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.140278 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.141137 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.141229 (+ 92us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.141279 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.141503 (+ 224us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.141682 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":82,"threads_started":1}
I20260923 23:10:29.142483 24792 tablet_service.cc:1467] Processing CreateTablet for tablet 16c759d24f8e45cd9346f16e00147356 (DEFAULT_TABLE table=test-table [id=e09344932cf045c281b39c3842c40ba0]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.142807 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 16c759d24f8e45cd9346f16e00147356. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.144292 24886 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Bootstrap starting.
I20260923 23:10:29.144834 24886 tablet_bootstrap.cc:654] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.144901 24887 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Bootstrap starting.
I20260923 23:10:29.145135 24886 log.cc:824] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:29.145459 24887 tablet_bootstrap.cc:654] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.145710 24887 log.cc:824] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:29.145902 24886 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: No bootstrap required, opened a new log
I20260923 23:10:29.145953 24886 ts_tablet_manager.cc:1397] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:29.145993 24889 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Bootstrap starting.
I20260923 23:10:29.146481 24889 tablet_bootstrap.cc:654] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.146718 24889 log.cc:824] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:29.147410 24887 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: No bootstrap required, opened a new log
I20260923 23:10:29.147478 24887 ts_tablet_manager.cc:1397] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:10:29.147680 24886 raft_consensus.cc:348] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.147850 24886 raft_consensus.cc:374] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.147886 24886 raft_consensus.cc:729] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94a7a36a360438bb73e8d5e34b63b03, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.147992 24886 consensus_queue.cc:260] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.148227 24886 ts_tablet_manager.cc:1428] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260923 23:10:29.148268 24578 heartbeater.cc:503] Master 127.16.122.190:37961 was elected leader, sending a full tablet report...
I20260923 23:10:29.149367 24887 raft_consensus.cc:348] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.149488 24887 raft_consensus.cc:374] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.149507 24887 raft_consensus.cc:729] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.149588 24887 consensus_queue.cc:260] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.149778 24887 ts_tablet_manager.cc:1428] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:29.149829 24718 heartbeater.cc:503] Master 127.16.122.190:37961 was elected leader, sending a full tablet report...
I20260923 23:10:29.150218 24889 tablet_bootstrap.cc:492] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: No bootstrap required, opened a new log
I20260923 23:10:29.150281 24889 ts_tablet_manager.cc:1397] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.000s
I20260923 23:10:29.151917 24889 raft_consensus.cc:348] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.152042 24889 raft_consensus.cc:374] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.152063 24889 raft_consensus.cc:729] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1045df71067405db80c3626137cb8a5, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.152166 24889 consensus_queue.cc:260] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.152376 24889 ts_tablet_manager.cc:1428] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:29.152418 24858 heartbeater.cc:503] Master 127.16.122.190:37961 was elected leader, sending a full tablet report...
I20260923 23:10:29.186568 24894 raft_consensus.cc:482] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:29.186674 24894 raft_consensus.cc:504] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.186954 24894 leader_election.cc:288] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113), a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125)
I20260923 23:10:29.189878 24751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.187161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34125 (local address 127.16.122.131:36237)
0923 23:10:29.187199 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.187210 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.187264 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.187591 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.187592 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.187594 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.187664 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.187666 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.188330 (+ 664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.188331 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.189098 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.189100 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.189635 (+ 535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189636 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.189717 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.189721 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.189766 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.189796 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":20}
I20260923 23:10:29.189896 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.187361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36237 (local address 127.16.122.129:34125)
0923 23:10:29.187449 (+ 88us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.187451 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.187463 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:29.187495 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.187496 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.187506 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.187583 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.187732 (+ 149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.188259 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189188 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.189561 (+ 373us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189618 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.189626 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.189788 (+ 162us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.189825 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260923 23:10:29.189942 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.187212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:54235 (local address 127.16.122.130:46113)
0923 23:10:29.187270 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.187273 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.187600 (+ 327us) server_negotiation.cc:407] Connection header received
0923 23:10:29.187649 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.187650 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.187659 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.187707 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.187866 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.188325 (+ 459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189240 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.189624 (+ 384us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189689 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.189697 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.189849 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.189884 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20260923 23:10:29.190207 24895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.187162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:46113 (local address 127.16.122.131:54235)
0923 23:10:29.187524 (+ 362us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.187534 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.187593 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.187716 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.187717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.187718 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.187799 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.187801 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.188398 (+ 597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.188399 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.189153 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.189156 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.189701 (+ 545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.189702 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.189765 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.189769 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.189836 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.189875 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":341,"mutex_wait_us":168,"thread_start_us":109,"threads_started":1}
I20260923 23:10:29.190197 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c759d24f8e45cd9346f16e00147356" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03" is_pre_election: true
I20260923 23:10:29.190342 24532 raft_consensus.cc:2454] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 0.
I20260923 23:10:29.190518 24672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c759d24f8e45cd9346f16e00147356" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff40c6dad56246ff83aac93b0904eba1" is_pre_election: true
I20260923 23:10:29.190562 24743 leader_election.cc:302] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94a7a36a360438bb73e8d5e34b63b03, b1045df71067405db80c3626137cb8a5; no voters:
I20260923 23:10:29.190661 24672 raft_consensus.cc:2454] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 0.
I20260923 23:10:29.190671 24894 raft_consensus.cc:2781] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:29.190750 24894 raft_consensus.cc:482] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:29.190778 24894 raft_consensus.cc:3037] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.191749 24894 raft_consensus.cc:504] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.191952 24894 leader_election.cc:288] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 election: Requested vote from peers ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113), a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125)
I20260923 23:10:29.192028 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c759d24f8e45cd9346f16e00147356" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03"
I20260923 23:10:29.192076 24672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "16c759d24f8e45cd9346f16e00147356" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff40c6dad56246ff83aac93b0904eba1"
I20260923 23:10:29.192113 24532 raft_consensus.cc:3037] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.192153 24672 raft_consensus.cc:3037] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.192849 24672 raft_consensus.cc:2454] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 1.
I20260923 23:10:29.192847 24532 raft_consensus.cc:2454] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 1.
I20260923 23:10:29.193058 24743 leader_election.cc:302] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94a7a36a360438bb73e8d5e34b63b03, b1045df71067405db80c3626137cb8a5; no voters:
I20260923 23:10:29.193132 24894 raft_consensus.cc:2781] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:29.193269 24894 raft_consensus.cc:686] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 1 LEADER]: Becoming Leader. State: Replica: b1045df71067405db80c3626137cb8a5, State: Running, Role: LEADER
I20260923 23:10:29.193353 24894 consensus_queue.cc:237] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } }
I20260923 23:10:29.194128 24374 catalog_manager.cc:5495] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1045df71067405db80c3626137cb8a5 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "b1045df71067405db80c3626137cb8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:29.200151 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.198064 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37206 (local address 127.16.122.131:40553)
0923 23:10:29.198111 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.198113 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.198141 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:10:29.198164 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.198165 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.198178 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.198223 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:29.198306 (+ 83us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.198759 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.199548 (+ 789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.199728 (+ 180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.199787 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.199896 (+ 109us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.200003 (+ 107us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.200063 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.200076 (+ 13us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.200092 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:10:29.201719 24532 raft_consensus.cc:1261] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Refusing update from remote peer b1045df71067405db80c3626137cb8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.201740 24672 raft_consensus.cc:1261] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Refusing update from remote peer b1045df71067405db80c3626137cb8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.202033 24896 consensus_queue.cc:1035] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.202155 24894 consensus_queue.cc:1035] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:10:29.203759 24719 tablet.cc:2369] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:29.205006 24901 mvcc.cc:205] Tried to move back new op lower bound from 7332679799608156160 to 7332679799576657920. Current Snapshot: MvccSnapshot[applied={T|T < 7332679799608156160 or (T in {7332679799608156160})}]
I20260923 23:10:29.206418 24374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.206507 24374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.207188 24374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Sending DeleteTablet for 3 replicas of tablet 16c759d24f8e45cd9346f16e00147356
I20260923 23:10:29.207531 24652 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c759d24f8e45cd9346f16e00147356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47998
I20260923 23:10:29.207566 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c759d24f8e45cd9346f16e00147356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40028
I20260923 23:10:29.207675 24374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260923 23:10:29.207769 24792 tablet_service.cc:1514] Processing DeleteTablet for tablet 16c759d24f8e45cd9346f16e00147356 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:37196
W20260923 23:10:29.207808 24374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:29.207945 24910 tablet_replica.cc:330] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: stopping tablet replica
I20260923 23:10:29.208038 24910 raft_consensus.cc:2229] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:29.208132 24910 raft_consensus.cc:2258] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.208895 24910 ts_tablet_manager.cc:1905] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.208985 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 23039b3aee7949259747296a813dacd3 (DEFAULT_TABLE table=test-table [id=e0950f9fc1fa47e5b206ff6be8031684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.209110 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23039b3aee7949259747296a813dacd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.210765 24886 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Bootstrap starting.
I20260923 23:10:29.210958 24792 tablet_service.cc:1467] Processing CreateTablet for tablet 23039b3aee7949259747296a813dacd3 (DEFAULT_TABLE table=test-table [id=e0950f9fc1fa47e5b206ff6be8031684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.210992 24652 tablet_service.cc:1467] Processing CreateTablet for tablet 23039b3aee7949259747296a813dacd3 (DEFAULT_TABLE table=test-table [id=e0950f9fc1fa47e5b206ff6be8031684]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.211122 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23039b3aee7949259747296a813dacd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.211126 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 23039b3aee7949259747296a813dacd3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.211212 24886 tablet_bootstrap.cc:654] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.212252 24889 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Bootstrap starting.
I20260923 23:10:29.212401 24910 ts_tablet_manager.cc:1918] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.212463 24910 log.cc:1196] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/16c759d24f8e45cd9346f16e00147356
I20260923 23:10:29.212672 24889 tablet_bootstrap.cc:654] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.212716 24910 ts_tablet_manager.cc:1939] T 16c759d24f8e45cd9346f16e00147356 P b1045df71067405db80c3626137cb8a5: Deleting consensus metadata
I20260923 23:10:29.212752 24887 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Bootstrap starting.
I20260923 23:10:29.213080 24360 catalog_manager.cc:4841] TS b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553): tablet 16c759d24f8e45cd9346f16e00147356 (table test-table [id=e09344932cf045c281b39c3842c40ba0]) successfully deleted
I20260923 23:10:29.213146 24887 tablet_bootstrap.cc:654] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.213743 24889 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: No bootstrap required, opened a new log
I20260923 23:10:29.213758 24887 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: No bootstrap required, opened a new log
I20260923 23:10:29.213786 24889 ts_tablet_manager.cc:1397] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:29.213799 24887 ts_tablet_manager.cc:1397] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:29.213886 24886 tablet_bootstrap.cc:492] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: No bootstrap required, opened a new log
I20260923 23:10:29.213920 24886 ts_tablet_manager.cc:1397] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:10:29.213913 24889 raft_consensus.cc:348] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.213940 24887 raft_consensus.cc:348] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.213986 24889 raft_consensus.cc:374] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.213999 24887 raft_consensus.cc:374] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.214018 24887 raft_consensus.cc:729] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.214007 24889 raft_consensus.cc:729] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1045df71067405db80c3626137cb8a5, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.214062 24889 consensus_queue.cc:260] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.214062 24887 consensus_queue.cc:260] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.214056 24886 raft_consensus.cc:348] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.214121 24886 raft_consensus.cc:374] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.214140 24886 raft_consensus.cc:729] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94a7a36a360438bb73e8d5e34b63b03, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.214147 24887 ts_tablet_manager.cc:1428] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:29.214157 24889 ts_tablet_manager.cc:1428] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.214176 24886 consensus_queue.cc:260] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.214250 24886 ts_tablet_manager.cc:1428] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.215131 24909 tablet_replica.cc:330] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: stopping tablet replica
I20260923 23:10:29.215245 24909 raft_consensus.cc:2229] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.215354 24909 raft_consensus.cc:2258] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.215524 24908 tablet_replica.cc:330] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: stopping tablet replica
I20260923 23:10:29.215631 24908 raft_consensus.cc:2229] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.215729 24908 raft_consensus.cc:2258] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.215808 24909 ts_tablet_manager.cc:1905] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.216070 24908 ts_tablet_manager.cc:1905] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.217255 24909 ts_tablet_manager.cc:1918] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.217324 24909 log.cc:1196] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/16c759d24f8e45cd9346f16e00147356
I20260923 23:10:29.217271 24908 ts_tablet_manager.cc:1918] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.217453 24908 log.cc:1196] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/16c759d24f8e45cd9346f16e00147356
I20260923 23:10:29.217585 24909 ts_tablet_manager.cc:1939] T 16c759d24f8e45cd9346f16e00147356 P a94a7a36a360438bb73e8d5e34b63b03: Deleting consensus metadata
I20260923 23:10:29.217697 24908 ts_tablet_manager.cc:1939] T 16c759d24f8e45cd9346f16e00147356 P ff40c6dad56246ff83aac93b0904eba1: Deleting consensus metadata
I20260923 23:10:29.217995 24359 catalog_manager.cc:4841] TS a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125): tablet 16c759d24f8e45cd9346f16e00147356 (table test-table [id=e09344932cf045c281b39c3842c40ba0]) successfully deleted
I20260923 23:10:29.218137 24360 catalog_manager.cc:4841] TS ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113): tablet 16c759d24f8e45cd9346f16e00147356 (table test-table [id=e09344932cf045c281b39c3842c40ba0]) successfully deleted
I20260923 23:10:29.233580 24894 raft_consensus.cc:482] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:29.233695 24894 raft_consensus.cc:504] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.233848 24894 leader_election.cc:288] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113)
I20260923 23:10:29.234058 24672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23039b3aee7949259747296a813dacd3" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff40c6dad56246ff83aac93b0904eba1" is_pre_election: true
I20260923 23:10:29.234100 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23039b3aee7949259747296a813dacd3" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03" is_pre_election: true
I20260923 23:10:29.234169 24672 raft_consensus.cc:2454] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 0.
I20260923 23:10:29.234186 24532 raft_consensus.cc:2454] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 0.
I20260923 23:10:29.234354 24743 leader_election.cc:302] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94a7a36a360438bb73e8d5e34b63b03, b1045df71067405db80c3626137cb8a5; no voters:
I20260923 23:10:29.234450 24894 raft_consensus.cc:2781] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:29.234479 24894 raft_consensus.cc:482] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:29.234498 24894 raft_consensus.cc:3037] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.235338 24894 raft_consensus.cc:504] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.235476 24894 leader_election.cc:288] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 election: Requested vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113)
I20260923 23:10:29.235661 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23039b3aee7949259747296a813dacd3" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03"
I20260923 23:10:29.235674 24672 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "23039b3aee7949259747296a813dacd3" candidate_uuid: "b1045df71067405db80c3626137cb8a5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ff40c6dad56246ff83aac93b0904eba1"
I20260923 23:10:29.235740 24532 raft_consensus.cc:3037] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.235743 24672 raft_consensus.cc:3037] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.236369 24672 raft_consensus.cc:2454] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 1.
I20260923 23:10:29.236442 24532 raft_consensus.cc:2454] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b1045df71067405db80c3626137cb8a5 in term 1.
I20260923 23:10:29.236533 24744 leader_election.cc:302] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1045df71067405db80c3626137cb8a5, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:29.236617 24894 raft_consensus.cc:2781] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:29.236663 24894 raft_consensus.cc:686] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 1 LEADER]: Becoming Leader. State: Replica: b1045df71067405db80c3626137cb8a5, State: Running, Role: LEADER
I20260923 23:10:29.236720 24894 consensus_queue.cc:237] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.237237 24375 catalog_manager.cc:5495] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 reported cstate change: term changed from 0 to 1, leader changed from <none> to b1045df71067405db80c3626137cb8a5 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "b1045df71067405db80c3626137cb8a5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } health_report { overall_health: HEALTHY } } }
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205028, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.190@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:29.278939 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.271842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39132 (local address 127.16.122.190:37961)
0923 23:10:29.271903 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.271905 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.272038 (+ 133us) server_negotiation.cc:407] Connection header received
0923 23:10:29.272112 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.272113 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.272121 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.272189 (+ 68us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.272815 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.273301 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.274326 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.274517 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.274575 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.276732 (+ 2157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:29.276734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:29.276734 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:29.276740 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:29.277949 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.278265 (+ 316us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.278266 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.278266 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.278318 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.278470 (+ 152us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.278471 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.278471 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.278573 (+ 102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:29.278626 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.278757 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.278842 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":36}
I20260923 23:10:29.279070 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.271774 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37961 (local address 127.0.0.1:39132)
0923 23:10:29.271950 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.271963 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.272050 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.272205 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.272206 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.272633 (+ 427us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:29.272714 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.272717 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.273385 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.273387 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.274246 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.274250 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.274592 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.274593 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.274677 (+ 84us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.275252 (+ 575us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:29.275256 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:29.276656 (+ 1400us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:29.278017 (+ 1361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.278025 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.278027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.278207 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.278369 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:29.278370 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:29.278370 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:29.278424 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:29.278621 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:29.278627 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:29.278671 (+ 44us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.278730 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.278942 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":144,"mutex_wait_us":66,"thread_start_us":81,"threads_started":1}
I20260923 23:10:29.281333 24375 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:39132:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260923 23:10:29.281513 24375 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:29.282847 24652 tablet_service.cc:1467] Processing CreateTablet for tablet 8eec14296898423884f094f3726eade4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fcd64a219f64632b4752d7e26ea52ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:29.282843 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 8eec14296898423884f094f3726eade4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fcd64a219f64632b4752d7e26ea52ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:29.282994 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eec14296898423884f094f3726eade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.283001 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eec14296898423884f094f3726eade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.284128 24887 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: Bootstrap starting.
I20260923 23:10:29.284475 24887 tablet_bootstrap.cc:654] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.284554 24886 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: Bootstrap starting.
I20260923 23:10:29.284924 24886 tablet_bootstrap.cc:654] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.284946 24792 tablet_service.cc:1467] Processing CreateTablet for tablet 8eec14296898423884f094f3726eade4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=2fcd64a219f64632b4752d7e26ea52ca]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260923 23:10:29.285084 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8eec14296898423884f094f3726eade4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.285522 24886 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: No bootstrap required, opened a new log
I20260923 23:10:29.285567 24886 ts_tablet_manager.cc:1397] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:29.285701 24886 raft_consensus.cc:348] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.285763 24886 raft_consensus.cc:374] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.285781 24886 raft_consensus.cc:729] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94a7a36a360438bb73e8d5e34b63b03, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.285828 24886 consensus_queue.cc:260] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.285913 24886 ts_tablet_manager.cc:1428] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.286028 24892 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.286088 24892 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.286423 24889 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: Bootstrap starting.
I20260923 23:10:29.286792 24889 tablet_bootstrap.cc:654] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.287467 24889 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: No bootstrap required, opened a new log
I20260923 23:10:29.287514 24889 ts_tablet_manager.cc:1397] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:29.287536 24887 tablet_bootstrap.cc:492] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: No bootstrap required, opened a new log
I20260923 23:10:29.287590 24887 ts_tablet_manager.cc:1397] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:10:29.287781 24889 raft_consensus.cc:348] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.287770 24887 raft_consensus.cc:348] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.287845 24889 raft_consensus.cc:374] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.287846 24887 raft_consensus.cc:374] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.287865 24889 raft_consensus.cc:729] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1045df71067405db80c3626137cb8a5, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.287868 24887 raft_consensus.cc:729] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.287909 24889 consensus_queue.cc:260] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.287915 24887 consensus_queue.cc:260] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.287997 24887 ts_tablet_manager.cc:1428] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: Time spent starting tablet: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:29.288000 24889 ts_tablet_manager.cc:1428] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:29.288077 24894 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.288136 24894 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.288127 24893 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.288206 24893 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: This TxnStatusTablet replica's current role is: FOLLOWER
W20260923 23:10:29.310534 24579 tablet.cc:2369] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:29.346545 24859 tablet.cc:2369] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:29.457086 24893 raft_consensus.cc:482] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:29.457203 24893 raft_consensus.cc:504] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.457526 24893 leader_election.cc:288] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553), a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125)
I20260923 23:10:29.460633 24925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.457709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:40553 (local address 127.16.122.130:56291)
0923 23:10:29.457846 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.457856 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.457923 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.458043 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.458044 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.458046 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.458116 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.458118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.458694 (+ 576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.458696 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.459652 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.459655 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.460356 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.460357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.460437 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.460442 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.460494 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.460535 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"mutex_wait_us":59,"thread_start_us":50,"threads_started":1}
I20260923 23:10:29.460625 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.457854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56291 (local address 127.16.122.131:40553)
0923 23:10:29.457919 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.457921 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.457929 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:29.457984 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.457985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.457996 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.458036 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.458172 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.458627 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.459773 (+ 1146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.460273 (+ 500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.460340 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.460349 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.460502 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.460560 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:10:29.461172 24924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.457709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34125 (local address 127.16.122.130:37253)
0923 23:10:29.457841 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.457851 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.457915 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.458048 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.458049 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.458051 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:29.458119 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.458121 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.458727 (+ 606us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.458728 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.459657 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.459660 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.460947 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.460947 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.461000 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.461004 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.461053 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.461090 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":102,"mutex_wait_us":3,"thread_start_us":56,"threads_started":1}
I20260923 23:10:29.461172 24883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.457771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37253 (local address 127.16.122.129:34125)
0923 23:10:29.457811 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.457814 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.457907 (+ 93us) server_negotiation.cc:407] Connection header received
0923 23:10:29.457977 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.457978 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.457987 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.458027 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:29.458175 (+ 148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.458653 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.460387 (+ 1734us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.460871 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.460948 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.460956 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.461071 (+ 115us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.461118 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:10:29.461542 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eec14296898423884f094f3726eade4" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5" is_pre_election: true
I20260923 23:10:29.461585 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eec14296898423884f094f3726eade4" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03" is_pre_election: true
I20260923 23:10:29.461695 24532 raft_consensus.cc:2454] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:29.461683 24812 raft_consensus.cc:2454] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:29.461862 24604 leader_election.cc:302] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1045df71067405db80c3626137cb8a5, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:29.462037 24893 raft_consensus.cc:2781] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:29.462076 24893 raft_consensus.cc:482] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:29.462093 24893 raft_consensus.cc:3037] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.463119 24893 raft_consensus.cc:504] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.463244 24893 leader_election.cc:288] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Requested vote from peers b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553), a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125)
I20260923 23:10:29.463408 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eec14296898423884f094f3726eade4" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5"
I20260923 23:10:29.463408 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8eec14296898423884f094f3726eade4" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03"
I20260923 23:10:29.463482 24812 raft_consensus.cc:3037] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.463482 24532 raft_consensus.cc:3037] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.464206 24812 raft_consensus.cc:2454] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:29.464205 24532 raft_consensus.cc:2454] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:29.464351 24603 leader_election.cc:302] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94a7a36a360438bb73e8d5e34b63b03, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:29.464433 24893 raft_consensus.cc:2781] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:29.464558 24893 raft_consensus.cc:686] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [term 1 LEADER]: Becoming Leader. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Running, Role: LEADER
I20260923 23:10:29.464622 24893 consensus_queue.cc:237] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } }
I20260923 23:10:29.464924 24926 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: TxnStatusTablet state changed. Reason: New leader ff40c6dad56246ff83aac93b0904eba1. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.464993 24926 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:29.465111 24928 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260923 23:10:29.465225 24928 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260923 23:10:29.465337 24373 catalog_manager.cc:5495] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } health_report { overall_health: HEALTHY } } }
I20260923 23:10:29.470713 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.468238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48008 (local address 127.16.122.130:46113)
0923 23:10:29.468273 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.468275 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.468302 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:10:29.468388 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.468388 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.468402 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.468454 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:29.468587 (+ 133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.469032 (+ 445us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.470062 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.470230 (+ 168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.470273 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.470455 (+ 182us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.470536 (+ 81us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.470580 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.470643 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.470658 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:29.470767 24920 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.468183 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:46113 (local address 127.0.0.1:48008)
0923 23:10:29.468259 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.468270 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.468318 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.468460 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.468461 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.468462 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:29.468532 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.468534 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.469094 (+ 560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.469095 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.469992 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.469995 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.470287 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.470288 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.470363 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.470379 (+ 16us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.470583 (+ 204us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.470588 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.470641 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.470690 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 23:10:29.471756 24532 raft_consensus.cc:1261] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.471792 24812 raft_consensus.cc:1261] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.472054 24926 consensus_queue.cc:1035] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.472170 24893 consensus_queue.cc:1035] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.472692 24892 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: TxnStatusTablet state changed. Reason: New leader ff40c6dad56246ff83aac93b0904eba1. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.472780 24892 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.472738 24894 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: TxnStatusTablet state changed. Reason: New leader ff40c6dad56246ff83aac93b0904eba1. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.472815 24894 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.473224 24900 mvcc.cc:205] Tried to move back new op lower bound from 7332679800714510336 to 7332679800687681536. Current Snapshot: MvccSnapshot[applied={T|T < 7332679800714510336}]
I20260923 23:10:29.473462 24926 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.473531 24926 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:29.473551 24899 mvcc.cc:205] Tried to move back new op lower bound from 7332679800714510336 to 7332679800687681536. Current Snapshot: MvccSnapshot[applied={T|T < 7332679800714510336}]
I20260923 23:10:29.473623 24894 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.473675 24894 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P b1045df71067405db80c3626137cb8a5: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.478905 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.476910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:44887 (local address 127.16.122.130:46113)
0923 23:10:29.476949 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.476951 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.476959 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:29.477014 (+ 55us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.477015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.477023 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.477068 (+ 45us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:29.477159 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.477598 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.478297 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.478489 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.478540 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.478687 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.478724 (+ 37us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.478781 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.478816 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.478836 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:10:29.478922 24864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.476852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:46113 (local address 127.16.122.131:44887)
0923 23:10:29.476911 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.476922 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.476961 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.477072 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.477072 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.477073 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:29.477116 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.477118 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.477664 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.477665 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.478229 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.478231 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.478544 (+ 313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.478545 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.478598 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.478610 (+ 12us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.478781 (+ 171us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.478782 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.478812 (+ 30us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.478852 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:10:29.480821 24672 raft_consensus.cc:1261] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Refusing update from remote peer b1045df71067405db80c3626137cb8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.480866 24531 raft_consensus.cc:1261] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Refusing update from remote peer b1045df71067405db80c3626137cb8a5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.481053 24894 consensus_queue.cc:1035] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.481143 24896 consensus_queue.cc:1035] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.483465 24893 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.483567 24893 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P ff40c6dad56246ff83aac93b0904eba1: This TxnStatusTablet replica's current role is: LEADER
I20260923 23:10:29.484433 24892 tablet_replica.cc:439] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } } peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } }
I20260923 23:10:29.484516 24892 tablet_replica.cc:442] T 8eec14296898423884f094f3726eade4 P a94a7a36a360438bb73e8d5e34b63b03: This TxnStatusTablet replica's current role is: FOLLOWER
I20260923 23:10:29.489801 24940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.486948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:40553 (local address 127.16.122.130:56921)
0923 23:10:29.487113 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:29.487124 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:29.487169 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:29.487337 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:29.487338 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:29.487342 (+ 4us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:29.487394 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.487395 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.488306 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.488306 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.489010 (+ 704us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:29.489013 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.489446 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.489447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:29.489492 (+ 45us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.489506 (+ 14us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.489655 (+ 149us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.489656 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:29.489683 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 23:10:29.489724 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":68,"threads_started":1}
I20260923 23:10:29.489794 24885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.487126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56921 (local address 127.16.122.131:40553)
0923 23:10:29.487204 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.487206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.487213 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:29.487245 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.487246 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.487255 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.487297 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:29.487458 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.488238 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.489081 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.489372 (+ 291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.489440 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.489576 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.489608 (+ 32us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.489653 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.489691 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.489713 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20260923 23:10:29.498144 24373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.498263 24373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39116:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.498816 24373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Sending DeleteTablet for 3 replicas of tablet 23039b3aee7949259747296a813dacd3
I20260923 23:10:29.499079 24792 tablet_service.cc:1514] Processing DeleteTablet for tablet 23039b3aee7949259747296a813dacd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:37196
I20260923 23:10:29.499075 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 23039b3aee7949259747296a813dacd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40028
I20260923 23:10:29.499168 24910 tablet_replica.cc:330] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: stopping tablet replica
I20260923 23:10:29.499176 24909 tablet_replica.cc:330] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: stopping tablet replica
I20260923 23:10:29.499223 24910 raft_consensus.cc:2229] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:29.499239 24909 raft_consensus.cc:2229] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.499269 24909 raft_consensus.cc:2258] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.499272 24910 raft_consensus.cc:2258] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.499662 24909 ts_tablet_manager.cc:1905] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.501441 24910 ts_tablet_manager.cc:1905] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.501451 24909 ts_tablet_manager.cc:1918] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:29.501497 24909 log.cc:1196] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/23039b3aee7949259747296a813dacd3
I20260923 23:10:29.501708 24909 ts_tablet_manager.cc:1939] T 23039b3aee7949259747296a813dacd3 P a94a7a36a360438bb73e8d5e34b63b03: Deleting consensus metadata
I20260923 23:10:29.502048 24359 catalog_manager.cc:4841] TS a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125): tablet 23039b3aee7949259747296a813dacd3 (table test-table [id=e0950f9fc1fa47e5b206ff6be8031684]) successfully deleted
I20260923 23:10:29.502825 24910 ts_tablet_manager.cc:1918] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:29.502876 24910 log.cc:1196] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/23039b3aee7949259747296a813dacd3
I20260923 23:10:29.503090 24910 ts_tablet_manager.cc:1939] T 23039b3aee7949259747296a813dacd3 P b1045df71067405db80c3626137cb8a5: Deleting consensus metadata
I20260923 23:10:29.503393 24360 catalog_manager.cc:4841] TS b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553): tablet 23039b3aee7949259747296a813dacd3 (table test-table [id=e0950f9fc1fa47e5b206ff6be8031684]) successfully deleted
I20260923 23:10:29.503571 24652 tablet_service.cc:1514] Processing DeleteTablet for tablet 23039b3aee7949259747296a813dacd3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47998
I20260923 23:10:29.503677 24908 tablet_replica.cc:330] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: stopping tablet replica
I20260923 23:10:29.503731 24908 raft_consensus.cc:2229] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.503762 24908 raft_consensus.cc:2258] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.503990 24908 ts_tablet_manager.cc:1905] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.505142 24908 ts_tablet_manager.cc:1918] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:29.505200 24908 log.cc:1196] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/23039b3aee7949259747296a813dacd3
I20260923 23:10:29.505366 24908 ts_tablet_manager.cc:1939] T 23039b3aee7949259747296a813dacd3 P ff40c6dad56246ff83aac93b0904eba1: Deleting consensus metadata
I20260923 23:10:29.505633 24360 catalog_manager.cc:4841] TS ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113): tablet 23039b3aee7949259747296a813dacd3 (table test-table [id=e0950f9fc1fa47e5b206ff6be8031684]) successfully deleted
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:10:29.526286 24426 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:29.520365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39148 (local address 127.16.122.190:37961)
0923 23:10:29.520427 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.520429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.520526 (+ 97us) server_negotiation.cc:407] Connection header received
0923 23:10:29.520584 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.520585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.520596 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.520677 (+ 81us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.521217 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.521997 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.522844 (+ 847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.523188 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.523258 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.526219 (+ 2961us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39148: BlockingRecv error: recv got EOF from 127.0.0.1:39148 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 23 23:10:29 dist-test-slave-95hh krb5kdc[24330](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205029, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM
I20260923 23:10:29.534209 24426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.527212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39154 (local address 127.16.122.190:37961)
0923 23:10:29.527265 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.527267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.527329 (+ 62us) server_negotiation.cc:407] Connection header received
0923 23:10:29.527400 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.527401 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.527408 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.527452 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:29.527970 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.528423 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.529372 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.529573 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.529624 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.531971 (+ 2347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:29.531972 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:29.531973 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:29.531980 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:29.533235 (+ 1255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.533541 (+ 306us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.533542 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.533542 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.533592 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:29.533750 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:29.533751 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:29.533751 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:29.533859 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:29.533941 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.534041 (+ 100us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.534112 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:29.535820 24373 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:29.536001 24373 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:29.537348 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 235503439d794436bdc07dddd852c443 (DEFAULT_TABLE table=test-table [id=c1abe113c7724d6a841a32cbc18c6814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.537433 24792 tablet_service.cc:1467] Processing CreateTablet for tablet 235503439d794436bdc07dddd852c443 (DEFAULT_TABLE table=test-table [id=c1abe113c7724d6a841a32cbc18c6814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.537479 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235503439d794436bdc07dddd852c443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.537479 24652 tablet_service.cc:1467] Processing CreateTablet for tablet 235503439d794436bdc07dddd852c443 (DEFAULT_TABLE table=test-table [id=c1abe113c7724d6a841a32cbc18c6814]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.537526 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235503439d794436bdc07dddd852c443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.537602 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 235503439d794436bdc07dddd852c443. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.538753 24889 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Bootstrap starting.
I20260923 23:10:29.538753 24886 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Bootstrap starting.
I20260923 23:10:29.538879 24887 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Bootstrap starting.
I20260923 23:10:29.539094 24889 tablet_bootstrap.cc:654] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.539144 24886 tablet_bootstrap.cc:654] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.539211 24887 tablet_bootstrap.cc:654] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.539959 24886 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: No bootstrap required, opened a new log
I20260923 23:10:29.539973 24887 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: No bootstrap required, opened a new log
I20260923 23:10:29.540011 24887 ts_tablet_manager.cc:1397] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:29.540027 24886 ts_tablet_manager.cc:1397] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:29.540113 24889 tablet_bootstrap.cc:492] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: No bootstrap required, opened a new log
I20260923 23:10:29.540150 24889 ts_tablet_manager.cc:1397] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:29.540148 24887 raft_consensus.cc:348] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540212 24887 raft_consensus.cc:374] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.540171 24886 raft_consensus.cc:348] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540231 24887 raft_consensus.cc:729] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.540236 24886 raft_consensus.cc:374] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.540256 24886 raft_consensus.cc:729] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94a7a36a360438bb73e8d5e34b63b03, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.540247 24889 raft_consensus.cc:348] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540269 24887 consensus_queue.cc:260] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540299 24889 raft_consensus.cc:374] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.540292 24886 consensus_queue.cc:260] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540318 24889 raft_consensus.cc:729] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1045df71067405db80c3626137cb8a5, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.540362 24887 ts_tablet_manager.cc:1428] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.540352 24889 consensus_queue.cc:260] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.540369 24886 ts_tablet_manager.cc:1428] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.540426 24889 ts_tablet_manager.cc:1428] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.845472 24893 raft_consensus.cc:482] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:29.845580 24893 raft_consensus.cc:504] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.845745 24893 leader_election.cc:288] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553)
I20260923 23:10:29.845990 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235503439d794436bdc07dddd852c443" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5" is_pre_election: true
I20260923 23:10:29.846029 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235503439d794436bdc07dddd852c443" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03" is_pre_election: true
I20260923 23:10:29.846084 24812 raft_consensus.cc:2454] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:29.846123 24532 raft_consensus.cc:2454] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:29.846269 24604 leader_election.cc:302] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1045df71067405db80c3626137cb8a5, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:29.846383 24893 raft_consensus.cc:2781] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:29.846427 24893 raft_consensus.cc:482] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:29.846447 24893 raft_consensus.cc:3037] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.847103 24893 raft_consensus.cc:504] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.847241 24893 leader_election.cc:288] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Requested vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553)
I20260923 23:10:29.847399 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235503439d794436bdc07dddd852c443" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5"
I20260923 23:10:29.847404 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "235503439d794436bdc07dddd852c443" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03"
I20260923 23:10:29.847464 24812 raft_consensus.cc:3037] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.847518 24532 raft_consensus.cc:3037] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:29.848059 24812 raft_consensus.cc:2454] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:29.848167 24532 raft_consensus.cc:2454] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:29.848219 24604 leader_election.cc:302] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1045df71067405db80c3626137cb8a5, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:29.848326 24893 raft_consensus.cc:2781] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:29.848393 24893 raft_consensus.cc:686] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 1 LEADER]: Becoming Leader. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Running, Role: LEADER
I20260923 23:10:29.848451 24893 consensus_queue.cc:237] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.849040 24373 catalog_manager.cc:5495] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:29.894234 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:29.892254 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46496 (local address 127.16.122.130:46113)
0923 23:10:29.892319 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:29.892320 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:29.892329 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:29.892352 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:29.892353 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:29.892362 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:29.892409 (+ 47us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:29.892533 (+ 124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.893008 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.893710 (+ 702us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:29.893900 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:29.893957 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:29.894048 (+ 91us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:29.894075 (+ 27us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:29.894109 (+ 34us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:29.894142 (+ 33us) server_negotiation.cc:299] Negotiation successful
0923 23:10:29.894156 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:10:29.895169 24812 raft_consensus.cc:1261] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.895162 24532 raft_consensus.cc:1261] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:29.895402 24926 consensus_queue.cc:1035] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.895558 24893 consensus_queue.cc:1035] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:29.898343 24374 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.898423 24374 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:29.898945 24374 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Sending DeleteTablet for 3 replicas of tablet 235503439d794436bdc07dddd852c443
I20260923 23:10:29.899190 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 235503439d794436bdc07dddd852c443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40028
I20260923 23:10:29.899232 24652 tablet_service.cc:1514] Processing DeleteTablet for tablet 235503439d794436bdc07dddd852c443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47998
I20260923 23:10:29.899250 24792 tablet_service.cc:1514] Processing DeleteTablet for tablet 235503439d794436bdc07dddd852c443 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:29 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:37196
I20260923 23:10:29.899319 24910 tablet_replica.cc:330] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: stopping tablet replica
I20260923 23:10:29.899322 24909 tablet_replica.cc:330] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: stopping tablet replica
I20260923 23:10:29.899381 24910 raft_consensus.cc:2229] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.899380 24909 raft_consensus.cc:2229] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:29.899423 24910 raft_consensus.cc:2258] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.899425 24909 raft_consensus.cc:2258] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.899489 24908 tablet_replica.cc:330] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: stopping tablet replica
I20260923 23:10:29.899536 24908 raft_consensus.cc:2229] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:29.899626 24908 raft_consensus.cc:2258] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:29.899765 24374 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:29.899890 24374 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:29.900019 24909 ts_tablet_manager.cc:1905] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.901369 24908 ts_tablet_manager.cc:1905] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.901373 24909 ts_tablet_manager.cc:1918] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.901429 24909 log.cc:1196] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/235503439d794436bdc07dddd852c443
I20260923 23:10:29.901628 24909 ts_tablet_manager.cc:1939] T 235503439d794436bdc07dddd852c443 P a94a7a36a360438bb73e8d5e34b63b03: Deleting consensus metadata
I20260923 23:10:29.902652 24359 catalog_manager.cc:4841] TS a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125): tablet 235503439d794436bdc07dddd852c443 (table test-table [id=c1abe113c7724d6a841a32cbc18c6814]) successfully deleted
I20260923 23:10:29.902734 24908 ts_tablet_manager.cc:1918] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.902787 24908 log.cc:1196] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/235503439d794436bdc07dddd852c443
I20260923 23:10:29.902981 24908 ts_tablet_manager.cc:1939] T 235503439d794436bdc07dddd852c443 P ff40c6dad56246ff83aac93b0904eba1: Deleting consensus metadata
I20260923 23:10:29.903229 24792 tablet_service.cc:1467] Processing CreateTablet for tablet 613edbcf6cbd4dc696646a90fb3305be (DEFAULT_TABLE table=test-table [id=9359df0dc6324e47aa04a5a77a03b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.903293 24512 tablet_service.cc:1467] Processing CreateTablet for tablet 613edbcf6cbd4dc696646a90fb3305be (DEFAULT_TABLE table=test-table [id=9359df0dc6324e47aa04a5a77a03b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.903324 24652 tablet_service.cc:1467] Processing CreateTablet for tablet 613edbcf6cbd4dc696646a90fb3305be (DEFAULT_TABLE table=test-table [id=9359df0dc6324e47aa04a5a77a03b001]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:29.903364 24792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613edbcf6cbd4dc696646a90fb3305be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.903393 24512 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613edbcf6cbd4dc696646a90fb3305be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.903402 24652 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 613edbcf6cbd4dc696646a90fb3305be. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:29.904479 24887 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Bootstrap starting.
I20260923 23:10:29.904590 24886 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: Bootstrap starting.
I20260923 23:10:29.904799 24887 tablet_bootstrap.cc:654] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.904955 24886 tablet_bootstrap.cc:654] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.905354 24889 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Bootstrap starting.
I20260923 23:10:29.905637 24887 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: No bootstrap required, opened a new log
I20260923 23:10:29.905659 24889 tablet_bootstrap.cc:654] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:29.905683 24887 ts_tablet_manager.cc:1397] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:29.905727 24910 ts_tablet_manager.cc:1905] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:29.905826 24887 raft_consensus.cc:348] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.905889 24887 raft_consensus.cc:374] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.905906 24887 raft_consensus.cc:729] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.905941 24887 consensus_queue.cc:260] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.906026 24887 ts_tablet_manager.cc:1428] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.906389 24889 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: No bootstrap required, opened a new log
I20260923 23:10:29.906430 24889 ts_tablet_manager.cc:1397] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Time spent bootstrapping tablet: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:29.906517 24889 raft_consensus.cc:348] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.906558 24889 raft_consensus.cc:374] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.906572 24889 raft_consensus.cc:729] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b1045df71067405db80c3626137cb8a5, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.906599 24889 consensus_queue.cc:260] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.906661 24889 ts_tablet_manager.cc:1428] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.906847 24886 tablet_bootstrap.cc:492] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: No bootstrap required, opened a new log
I20260923 23:10:29.906875 24886 ts_tablet_manager.cc:1397] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:29.906976 24886 raft_consensus.cc:348] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.906977 24910 ts_tablet_manager.cc:1918] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:29.907016 24886 raft_consensus.cc:374] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:29.907053 24886 raft_consensus.cc:729] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a94a7a36a360438bb73e8d5e34b63b03, State: Initialized, Role: FOLLOWER
I20260923 23:10:29.907056 24910 log.cc:1196] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/235503439d794436bdc07dddd852c443
I20260923 23:10:29.907092 24886 consensus_queue.cc:260] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:29.907174 24886 ts_tablet_manager.cc:1428] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:29.907256 24910 ts_tablet_manager.cc:1939] T 235503439d794436bdc07dddd852c443 P b1045df71067405db80c3626137cb8a5: Deleting consensus metadata
I20260923 23:10:29.907364 24360 catalog_manager.cc:4841] TS ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113): tablet 235503439d794436bdc07dddd852c443 (table test-table [id=c1abe113c7724d6a841a32cbc18c6814]) successfully deleted
I20260923 23:10:29.907931 24360 catalog_manager.cc:4841] TS b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553): tablet 235503439d794436bdc07dddd852c443 (table test-table [id=c1abe113c7724d6a841a32cbc18c6814]) successfully deleted
I20260923 23:10:30.154088 24926 raft_consensus.cc:482] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:30.154196 24926 raft_consensus.cc:504] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:30.154363 24926 leader_election.cc:288] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553)
I20260923 23:10:30.154589 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613edbcf6cbd4dc696646a90fb3305be" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5" is_pre_election: true
I20260923 23:10:30.154608 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613edbcf6cbd4dc696646a90fb3305be" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03" is_pre_election: true
I20260923 23:10:30.154681 24812 raft_consensus.cc:2454] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:30.154702 24532 raft_consensus.cc:2454] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 0.
I20260923 23:10:30.154819 24604 leader_election.cc:302] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b1045df71067405db80c3626137cb8a5, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:30.154930 24926 raft_consensus.cc:2781] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:30.154968 24926 raft_consensus.cc:482] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:30.154981 24926 raft_consensus.cc:3037] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:30.155699 24926 raft_consensus.cc:504] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:30.155807 24926 leader_election.cc:288] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Requested vote from peers a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125), b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553)
I20260923 23:10:30.155947 24532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613edbcf6cbd4dc696646a90fb3305be" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a94a7a36a360438bb73e8d5e34b63b03"
I20260923 23:10:30.155947 24812 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "613edbcf6cbd4dc696646a90fb3305be" candidate_uuid: "ff40c6dad56246ff83aac93b0904eba1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b1045df71067405db80c3626137cb8a5"
I20260923 23:10:30.156016 24812 raft_consensus.cc:3037] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:30.156016 24532 raft_consensus.cc:3037] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:30.156563 24532 raft_consensus.cc:2454] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:30.156637 24812 raft_consensus.cc:2454] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff40c6dad56246ff83aac93b0904eba1 in term 1.
I20260923 23:10:30.156719 24603 leader_election.cc:302] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a94a7a36a360438bb73e8d5e34b63b03, ff40c6dad56246ff83aac93b0904eba1; no voters:
I20260923 23:10:30.156808 24926 raft_consensus.cc:2781] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:30.156855 24926 raft_consensus.cc:686] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 1 LEADER]: Becoming Leader. State: Replica: ff40c6dad56246ff83aac93b0904eba1, State: Running, Role: LEADER
I20260923 23:10:30.156911 24926 consensus_queue.cc:237] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } }
I20260923 23:10:30.157478 24373 catalog_manager.cc:5495] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 reported cstate change: term changed from 0 to 1, leader changed from <none> to ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "ff40c6dad56246ff83aac93b0904eba1" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ff40c6dad56246ff83aac93b0904eba1" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 46113 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:30.192582 24882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:30.190617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:35563 (local address 127.16.122.130:46113)
0923 23:10:30.190654 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:30.190657 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:30.190706 (+ 49us) server_negotiation.cc:407] Connection header received
0923 23:10:30.190752 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:30.190753 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:30.190760 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:30.190803 (+ 43us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:30.190889 (+ 86us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:30.191377 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:30.192065 (+ 688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:30.192252 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:30.192302 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:30.192381 (+ 79us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:30.192403 (+ 22us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:30.192432 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:30.192470 (+ 38us) server_negotiation.cc:299] Negotiation successful
0923 23:10:30.192484 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:10:30.192595 24969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:30.190537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:46113 (local address 127.16.122.130:35563)
0923 23:10:30.190658 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:30.190667 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:30.190717 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:30.190812 (+ 95us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:30.190813 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:30.190814 (+ 1us) client_negotiation.cc:189] Negotiated authn=TOKEN
0923 23:10:30.190856 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:30.190858 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:30.191424 (+ 566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:30.191424 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:30.191991 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:30.191993 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:30.192307 (+ 314us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:30.192308 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:30.192341 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:30.192345 (+ 4us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0923 23:10:30.192440 (+ 95us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0923 23:10:30.192441 (+ 1us) client_negotiation.cc:769] Sending connection context
0923 23:10:30.192469 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 23:10:30.192502 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":56,"threads_started":1}
I20260923 23:10:30.194187 24812 raft_consensus.cc:1261] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:30.194195 24532 raft_consensus.cc:1261] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Refusing update from remote peer ff40c6dad56246ff83aac93b0904eba1: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:30.194397 24937 consensus_queue.cc:1035] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a94a7a36a360438bb73e8d5e34b63b03" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34125 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:30.194551 24937 consensus_queue.cc:1035] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b1045df71067405db80c3626137cb8a5" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 40553 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:30.204169 24373 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:30.204268 24373 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:39154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:30.204806 24373 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f9355750eaf64779adc1a6ad15c718a7: Sending DeleteTablet for 3 replicas of tablet 613edbcf6cbd4dc696646a90fb3305be
I20260923 23:10:30.205046 24512 tablet_service.cc:1514] Processing DeleteTablet for tablet 613edbcf6cbd4dc696646a90fb3305be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:30 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40028
I20260923 23:10:30.205086 24792 tablet_service.cc:1514] Processing DeleteTablet for tablet 613edbcf6cbd4dc696646a90fb3305be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:30 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:37196
I20260923 23:10:30.205142 24909 tablet_replica.cc:330] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03: stopping tablet replica
I20260923 23:10:30.205157 24910 tablet_replica.cc:330] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: stopping tablet replica
I20260923 23:10:30.205188 24909 raft_consensus.cc:2229] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:30.205196 24910 raft_consensus.cc:2229] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:30.205217 24909 raft_consensus.cc:2258] T 613edbcf6cbd4dc696646a90fb3305be P a94a7a36a360438bb73e8d5e34b63b03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:30.205222 24910 raft_consensus.cc:2258] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:30.205294 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 24447
I20260923 23:10:30.206571 24652 tablet_service.cc:1514] Processing DeleteTablet for tablet 613edbcf6cbd4dc696646a90fb3305be with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:30 UTC) from {username='oryx', principal='oryx/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47998
I20260923 23:10:30.206687 24908 tablet_replica.cc:330] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: stopping tablet replica
I20260923 23:10:30.206743 24908 raft_consensus.cc:2229] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:30.206806 24908 raft_consensus.cc:2258] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:30.207119 24908 ts_tablet_manager.cc:1905] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:30.207119 24910 ts_tablet_manager.cc:1905] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:30.208408 24908 ts_tablet_manager.cc:1918] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:30.208452 24908 log.cc:1196] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/613edbcf6cbd4dc696646a90fb3305be
I20260923 23:10:30.208632 24908 ts_tablet_manager.cc:1939] T 613edbcf6cbd4dc696646a90fb3305be P ff40c6dad56246ff83aac93b0904eba1: Deleting consensus metadata
I20260923 23:10:30.208616 24910 ts_tablet_manager.cc:1918] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260923 23:10:30.208663 24910 log.cc:1196] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/613edbcf6cbd4dc696646a90fb3305be
I20260923 23:10:30.208877 24910 ts_tablet_manager.cc:1939] T 613edbcf6cbd4dc696646a90fb3305be P b1045df71067405db80c3626137cb8a5: Deleting consensus metadata
I20260923 23:10:30.209012 24360 catalog_manager.cc:4841] TS ff40c6dad56246ff83aac93b0904eba1 (127.16.122.130:46113): tablet 613edbcf6cbd4dc696646a90fb3305be (table test-table [id=9359df0dc6324e47aa04a5a77a03b001]) successfully deleted
I20260923 23:10:30.209246 24360 catalog_manager.cc:4841] TS b1045df71067405db80c3626137cb8a5 (127.16.122.131:40553): tablet 613edbcf6cbd4dc696646a90fb3305be (table test-table [id=9359df0dc6324e47aa04a5a77a03b001]) successfully deleted
W20260923 23:10:30.210781 24359 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:34125 (error 108)
I20260923 23:10:30.210953 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 24587
I20260923 23:10:30.212276 24974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:30.212090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34125 (local address 127.0.0.1:48302)
0923 23:10:30.212192 (+ 102us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:30.212235 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:34125: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":46,"threads_started":1}
W20260923 23:10:30.212400 24359 catalog_manager.cc:4584] TS a94a7a36a360438bb73e8d5e34b63b03 (127.16.122.129:34125): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 613edbcf6cbd4dc696646a90fb3305be: Network error: Client connection negotiation failed: client connection to 127.16.122.129:34125: connect: Connection refused (error 111)
I20260923 23:10:30.216641 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 24727
I20260923 23:10:30.221448 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 24343
2026-09-23T23:10:30Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (2816 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:30 dist-test-slave-95hh krb5kdc[24977](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:30 dist-test-slave-95hh krb5kdc[24977](info): set up 2 sockets
Sep 23 23:10:30 dist-test-slave-95hh krb5kdc[24977](info): commencing operation
krb5kdc: starting...
W20260923 23:10:31.171996 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.001s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:31 dist-test-slave-95hh krb5kdc[24977](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205031, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:31.281241 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:38805
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:37461
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:38805
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:31.358994 24990 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:31.359259 24990 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:31.359282 24990 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:31.360394 24990 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:31.360425 24990 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:31.360437 24990 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:31.360450 24990 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:31.360459 24990 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:31.361872 24990 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:37461
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:38805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:38805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.24990
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:31.362073 24990 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:31.362259 24990 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:31.366969 24996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:31.367175 24990 server_base.cc:1034] running on GCE node
W20260923 23:10:31.366972 24995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:31.367002 24998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:31.367550 24990 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:31.367836 24990 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:31.368988 24990 hybrid_clock.cc:648] HybridClock initialized: now 1790205031368963 us; error 37 us; skew 500 ppm
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-09-23T23:10:31Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (1157 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:31 dist-test-slave-95hh krb5kdc[25003](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:31 dist-test-slave-95hh krb5kdc[25003](info): set up 2 sockets
Sep 23 23:10:31 dist-test-slave-95hh krb5kdc[25003](info): commencing operation
krb5kdc: starting...
W20260923 23:10:32.323880 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:32Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:32.431970 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35905
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:32.512326 25016 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:32.512547 25016 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:32.512570 25016 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:32.513770 25016 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:32.513813 25016 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:32.513825 25016 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:32.513836 25016 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:32.513847 25016 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:32.515645 25016 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25016
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:32.515882 25016 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:32.516100 25016 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:32.520676 25022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:32.520730 25024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:32.520750 25016 server_base.cc:1034] running on GCE node
W20260923 23:10:32.520670 25021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:32.521164 25016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:32.521425 25016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:32.522617 25016 hybrid_clock.cc:648] HybridClock initialized: now 1790205032522595 us; error 36 us; skew 500 ppm
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:32.535332 25016 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:32.535853 25016 webserver.cc:460] Webserver started at http://127.16.122.190:46281/ using document root <none> and password file <none>
I20260923 23:10:32.536072 25016 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:32.536113 25016 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:32.536212 25016 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:32.537273 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "c63b7c663e674c73b99b8c395e13599d"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "63c694848630e5bba1ebcc889dbe01e6"
server_key_iv: "676deb24dd141f3957b0a9413e0eac5d"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.537566 25016 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "c63b7c663e674c73b99b8c395e13599d"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "63c694848630e5bba1ebcc889dbe01e6"
server_key_iv: "676deb24dd141f3957b0a9413e0eac5d"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.538784 25016 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:32.539482 25031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.539664 25016 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.539752 25016 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "c63b7c663e674c73b99b8c395e13599d"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "63c694848630e5bba1ebcc889dbe01e6"
server_key_iv: "676deb24dd141f3957b0a9413e0eac5d"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.539808 25016 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:32.557109 25016 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:32.559180 25016 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:32.559291 25016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:32.562887 25016 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:35905
I20260923 23:10:32.562933 25083 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:35905 every 8 connection(s)
I20260923 23:10:32.563282 25016 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:32.564107 25084 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:32.566252 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25016
I20260923 23:10:32.566341 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:32.566425 25084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Bootstrap starting.
I20260923 23:10:32.566494 16874 external_mini_cluster.cc:1428] Setting key 49ecbeaeac1acf918bc1e6a2b7942bcc
I20260923 23:10:32.567071 25084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:32.567378 25084 log.cc:824] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:32.568032 25084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: No bootstrap required, opened a new log
I20260923 23:10:32.569988 25084 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:32.570164 25084 raft_consensus.cc:374] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:32.570195 25084 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63b7c663e674c73b99b8c395e13599d, State: Initialized, Role: FOLLOWER
I20260923 23:10:32.570317 25084 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:32.570386 25084 raft_consensus.cc:388] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:32.570428 25084 raft_consensus.cc:482] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:32.570508 25084 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:32.570986 25084 raft_consensus.cc:504] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:32.571116 25084 leader_election.cc:302] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c63b7c663e674c73b99b8c395e13599d; no voters:
I20260923 23:10:32.571285 25084 leader_election.cc:288] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:32.571308 25089 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:32.571504 25089 raft_consensus.cc:686] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 LEADER]: Becoming Leader. State: Replica: c63b7c663e674c73b99b8c395e13599d, State: Running, Role: LEADER
I20260923 23:10:32.571637 25084 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:32.571691 25089 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:32.572057 25090 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63b7c663e674c73b99b8c395e13599d. Latest consensus state: current_term: 1 leader_uuid: "c63b7c663e674c73b99b8c395e13599d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } } }
I20260923 23:10:32.572204 25090 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:32.573192 25091 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c63b7c663e674c73b99b8c395e13599d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } } }
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:32.573626 25091 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: This master's current role is: LEADER
W20260923 23:10:32.573788 25105 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:10:32.573844 25105 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:10:32.573889 25095 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:32.574064 25095 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:32.575805 25095 catalog_manager.cc:1347] Generated new cluster ID: 5b95f26033e04a38b060d9f8766e7688
I20260923 23:10:32.575865 25095 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:32.577415 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:32.566970 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57730 (local address 127.16.122.190:35905)
0923 23:10:32.567142 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:32.567151 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:32.567184 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:10:32.567637 (+ 453us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:32.567644 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:32.567693 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:32.567807 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:32.568552 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.569431 (+ 879us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.570472 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.570948 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.571068 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:32.574128 (+ 3060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:32.574133 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:32.574133 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:32.574145 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:32.576037 (+ 1892us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.576344 (+ 307us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.576348 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.576348 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.576436 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.576607 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.576608 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.576608 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.576833 (+ 225us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:32.576978 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:32.577081 (+ 103us) server_negotiation.cc:299] Negotiation successful
0923 23:10:32.577211 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260923 23:10:32.590122 25095 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:32.590838 25095 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:32.604732 25095 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Generated new TSK 0
I20260923 23:10:32.604943 25095 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:32.637571 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:32.717357 25110 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:32.717545 25110 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:32.717567 25110 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:32.718698 25110 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:32.718736 25110 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:32.718763 25110 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:32.720178 25110 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25110
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:32.720389 25110 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:32.720578 25110 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:32.725123 25115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:32.725133 25118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:32.725142 25116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:32.725284 25110 server_base.cc:1034] running on GCE node
I20260923 23:10:32.725553 25110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:32.725782 25110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:32.726944 25110 hybrid_clock.cc:648] HybridClock initialized: now 1790205032726929 us; error 29 us; skew 500 ppm
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:32.729387 25110 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:32.729806 25110 webserver.cc:460] Webserver started at http://127.16.122.129:40877/ using document root <none> and password file <none>
I20260923 23:10:32.730000 25110 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:32.730041 25110 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:32.730158 25110 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:32.731192 25110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "b228f2d8984948bc963073a6b13ac480"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "75b97c11669f13b4f9ea22dc2427718b"
server_key_iv: "f53167d8171f87b093b7f165dbb6aca4"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.731505 25110 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "b228f2d8984948bc963073a6b13ac480"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "75b97c11669f13b4f9ea22dc2427718b"
server_key_iv: "f53167d8171f87b093b7f165dbb6aca4"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.732733 25110 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:32.733353 25125 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.733500 25110 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:32.733577 25110 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "b228f2d8984948bc963073a6b13ac480"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "75b97c11669f13b4f9ea22dc2427718b"
server_key_iv: "f53167d8171f87b093b7f165dbb6aca4"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.733659 25110 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:32.755527 25110 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:32.757686 25110 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:32.757824 25110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:32.758038 25110 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:32.758328 25110 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:32.758365 25110 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.758394 25110 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:32.758416 25110 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.763571 25110 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:40235
I20260923 23:10:32.763641 25238 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:40235 every 8 connection(s)
I20260923 23:10:32.763936 25110 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:32.772490 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:32.764619 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:45269 (local address 127.16.122.190:35905)
0923 23:10:32.764662 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:32.764664 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:32.764820 (+ 156us) server_negotiation.cc:407] Connection header received
0923 23:10:32.765291 (+ 471us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:32.765292 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:32.765302 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:32.765361 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:32.766259 (+ 898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.766778 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.767744 (+ 966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.767951 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.768004 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:32.770001 (+ 1997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:32.770003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:32.770003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:32.770010 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:32.771154 (+ 1144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.771506 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.771507 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.771507 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.771721 (+ 214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.771938 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.771939 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.771939 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.772054 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:32.772103 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:32.772344 (+ 241us) server_negotiation.cc:299] Negotiation successful
0923 23:10:32.772422 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:32.772619 25241 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:32.764566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:45269)
0923 23:10:32.764709 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:32.764750 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:32.765202 (+ 452us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:32.765415 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:32.765418 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:32.765964 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:32.766158 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:32.766163 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.766848 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.766849 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:32.767671 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:32.767673 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.768011 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.768012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:32.768104 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:32.768561 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:32.768569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:32.769910 (+ 1341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:32.771213 (+ 1303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:32.771219 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:32.771220 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:32.771421 (+ 201us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:32.771785 (+ 364us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:32.771786 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:32.771786 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:32.771888 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:32.772097 (+ 209us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:32.772099 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:32.772219 (+ 120us) client_negotiation.cc:769] Sending connection context
0923 23:10:32.772313 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:10:32.772438 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260923 23:10:32.772720 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25110
I20260923 23:10:32.772838 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:32.773022 16874 external_mini_cluster.cc:1428] Setting key 5f93563b4cb5399ed3c008f60e0d5ba1
I20260923 23:10:32.773139 25239 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35905
I20260923 23:10:32.773253 25239 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:32.773479 25239 heartbeater.cc:511] Master 127.16.122.190:35905 requested a full tablet report, sending...
I20260923 23:10:32.773968 25048 ts_manager.cc:194] Registered new tserver with Master: b228f2d8984948bc963073a6b13ac480 (127.16.122.129:40235)
I20260923 23:10:32.774549 25048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:45269
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:32.794807 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:32.875818 25244 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:32.876020 25244 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:32.876042 25244 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:32.877171 25244 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:32.877211 25244 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:32.877236 25244 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:32.878611 25244 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25244
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:32.878842 25244 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:32.879068 25244 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:32.883634 25250 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:32.883671 25252 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:32.883698 25249 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:32.883891 25244 server_base.cc:1034] running on GCE node
I20260923 23:10:32.884106 25244 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:32.884318 25244 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:32.885448 25244 hybrid_clock.cc:648] HybridClock initialized: now 1790205032885435 us; error 28 us; skew 500 ppm
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:32.887830 25244 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:32.888229 25244 webserver.cc:460] Webserver started at http://127.16.122.130:37927/ using document root <none> and password file <none>
I20260923 23:10:32.888430 25244 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:32.888477 25244 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:32.888605 25244 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:32.889601 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "c1364130c0434c4da9f95bfdc05b5d4e"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "0968d3dc13890a5c86e8e10803ebbc5b"
server_key_iv: "8a6743e763d044d09c1b8529413b1fae"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.889917 25244 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "c1364130c0434c4da9f95bfdc05b5d4e"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "0968d3dc13890a5c86e8e10803ebbc5b"
server_key_iv: "8a6743e763d044d09c1b8529413b1fae"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.891137 25244 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:10:32.891764 25259 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.891906 25244 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:32.891983 25244 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "c1364130c0434c4da9f95bfdc05b5d4e"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "0968d3dc13890a5c86e8e10803ebbc5b"
server_key_iv: "8a6743e763d044d09c1b8529413b1fae"
server_key_version: "encryptionkey@0"
I20260923 23:10:32.892046 25244 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:32.912068 25244 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:32.913834 25244 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:32.913967 25244 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:32.914182 25244 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:32.914469 25244 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:32.914503 25244 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.914530 25244 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:32.914553 25244 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:32.919963 25244 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:38905
I20260923 23:10:32.920013 25372 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:38905 every 8 connection(s)
I20260923 23:10:32.920310 25244 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:32.920547 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25244
I20260923 23:10:32.920661 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:32.920822 16874 external_mini_cluster.cc:1428] Setting key 2342f9f639a32076acc2cb2229c19671
Sep 23 23:10:32 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205032, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
I20260923 23:10:32.929406 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:32.920973 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56207 (local address 127.16.122.190:35905)
0923 23:10:32.921052 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:32.921055 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:32.921236 (+ 181us) server_negotiation.cc:407] Connection header received
0923 23:10:32.921752 (+ 516us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:32.921754 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:32.921764 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:32.921821 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:32.922760 (+ 939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.923280 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.924224 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.924471 (+ 247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.924532 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:32.926810 (+ 2278us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:32.926811 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:32.926812 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:32.926818 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:32.928042 (+ 1224us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.928476 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.928477 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.928477 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.928521 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:32.928826 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:32.928826 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:32.928826 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:32.928919 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:32.928963 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:32.929265 (+ 302us) server_negotiation.cc:299] Negotiation successful
0923 23:10:32.929340 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260923 23:10:32.929634 25375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:32.920947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:56207)
0923 23:10:32.921118 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:32.921160 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:32.921663 (+ 503us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:32.921897 (+ 234us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:32.921902 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:32.922470 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:32.922665 (+ 195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:32.922670 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.923357 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.923358 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:32.924143 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:32.924146 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:32.924549 (+ 403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:32.924550 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:32.924685 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:32.925169 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:32.925177 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:32.926708 (+ 1531us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:32.928121 (+ 1413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:32.928128 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:32.928130 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:32.928407 (+ 277us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:32.928583 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:32.928584 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:32.928584 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:32.928763 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:32.928973 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:32.928978 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:32.929134 (+ 156us) client_negotiation.cc:769] Sending connection context
0923 23:10:32.929250 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 23:10:32.929415 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":89,"threads_started":1}
I20260923 23:10:32.930056 25373 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35905
I20260923 23:10:32.930155 25373 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:32.930330 25373 heartbeater.cc:511] Master 127.16.122.190:35905 requested a full tablet report, sending...
I20260923 23:10:32.930680 25048 ts_manager.cc:194] Registered new tserver with Master: c1364130c0434c4da9f95bfdc05b5d4e (127.16.122.130:38905)
I20260923 23:10:32.931070 25048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:56207
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:32.945839 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:33.025532 25378 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:33.025781 25378 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:33.025813 25378 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:33.027006 25378 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:33.027066 25378 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:33.027103 25378 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:33.028563 25378 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25378
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:33.028812 25378 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:33.029026 25378 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:33.033811 25386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.033843 25383 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.033811 25384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.034008 25378 server_base.cc:1034] running on GCE node
I20260923 23:10:33.034298 25378 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:33.034513 25378 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:33.035655 25378 hybrid_clock.cc:648] HybridClock initialized: now 1790205033035630 us; error 29 us; skew 500 ppm
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:33.038196 25378 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:33.038630 25378 webserver.cc:460] Webserver started at http://127.16.122.131:38125/ using document root <none> and password file <none>
I20260923 23:10:33.038844 25378 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:33.038888 25378 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:33.039008 25378 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:33.040107 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "08d7e999df0b491a9cdec853e3994b18"
format_stamp: "Formatted at 2026-09-23 23:10:33 on dist-test-slave-95hh"
server_key: "9850753c9da724062ed6316334b6115b"
server_key_iv: "3d6e7c95f923897132eefdd5583f5a4e"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.040434 25378 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "08d7e999df0b491a9cdec853e3994b18"
format_stamp: "Formatted at 2026-09-23 23:10:33 on dist-test-slave-95hh"
server_key: "9850753c9da724062ed6316334b6115b"
server_key_iv: "3d6e7c95f923897132eefdd5583f5a4e"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.041687 25378 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:10:33.042388 25393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.042537 25378 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:33.042621 25378 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "08d7e999df0b491a9cdec853e3994b18"
format_stamp: "Formatted at 2026-09-23 23:10:33 on dist-test-slave-95hh"
server_key: "9850753c9da724062ed6316334b6115b"
server_key_iv: "3d6e7c95f923897132eefdd5583f5a4e"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.042697 25378 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:33.081575 25378 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:33.083359 25378 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:33.083487 25378 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:33.083703 25378 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:33.083994 25378 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:33.084033 25378 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.084062 25378 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:33.084085 25378 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.089447 25378 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:44467
I20260923 23:10:33.089516 25506 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:44467 every 8 connection(s)
I20260923 23:10:33.089804 25378 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:33.090730 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25378
I20260923 23:10:33.090860 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:33.091065 16874 external_mini_cluster.cc:1428] Setting key b27a5f16b78d0e2c04fc1b491e9c3b71
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:33.098249 25087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.090453 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:51427 (local address 127.16.122.190:35905)
0923 23:10:33.090506 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.090508 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.090748 (+ 240us) server_negotiation.cc:407] Connection header received
0923 23:10:33.091405 (+ 657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.091406 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.091417 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.091474 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.092444 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.092931 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.093831 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.094002 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.094042 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.096005 (+ 1963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:33.096006 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:33.096006 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:33.096012 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:33.097269 (+ 1257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:33.097565 (+ 296us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:33.097565 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:33.097566 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:33.097617 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:33.097787 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:33.097787 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:33.097787 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:33.097885 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:33.097952 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:33.098127 (+ 175us) server_negotiation.cc:299] Negotiation successful
0923 23:10:33.098185 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:10:33.098419 25509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.090438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:51427)
0923 23:10:33.090584 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.090631 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.091306 (+ 675us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.091545 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.091549 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.092173 (+ 624us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.092372 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.092377 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.092994 (+ 617us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.092994 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.093774 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.093776 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.094055 (+ 279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.094056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.094151 (+ 95us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.094591 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.094600 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.095934 (+ 1334us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:33.097326 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:33.097329 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:33.097330 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:33.097517 (+ 187us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:33.097665 (+ 148us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:33.097665 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:33.097665 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:33.097748 (+ 83us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:33.097935 (+ 187us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:33.097937 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:33.098042 (+ 105us) client_negotiation.cc:769] Sending connection context
0923 23:10:33.098114 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 23:10:33.098227 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":73,"threads_started":1}
I20260923 23:10:33.098867 25507 heartbeater.cc:347] Connected to a master server at 127.16.122.190:35905
I20260923 23:10:33.098959 25507 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:33.099166 25507 heartbeater.cc:511] Master 127.16.122.190:35905 requested a full tablet report, sending...
I20260923 23:10:33.099491 25048 ts_manager.cc:194] Registered new tserver with Master: 08d7e999df0b491a9cdec853e3994b18 (127.16.122.131:44467)
I20260923 23:10:33.099835 25048 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:51427
I20260923 23:10:33.102420 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
Entry for principal oryx/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab.
WARNING: no policy specified for oryx/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
Entry for principal oryx/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab.
WARNING: no policy specified for oryx/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
Entry for principal oryx/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab.
I20260923 23:10:33.167203 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25110
I20260923 23:10:33.171252 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25244
I20260923 23:10:33.175201 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25378
I20260923 23:10:33.178782 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25016
I20260923 23:10:33.183109 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35905
--webserver_interface=127.16.122.190
--webserver_port=46281
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:35905
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:33.260977 25516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:33.261198 25516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:33.261219 25516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:33.262372 25516 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:33.262414 25516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:33.262427 25516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:33.262439 25516 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:33.262452 25516 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:33.263952 25516 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:35905
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:35905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=46281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25516
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:33.264187 25516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:33.264397 25516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:33.268971 25524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.268994 25521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.268970 25522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.269163 25516 server_base.cc:1034] running on GCE node
I20260923 23:10:33.269383 25516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:33.269671 25516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:33.270844 25516 hybrid_clock.cc:648] HybridClock initialized: now 1790205033270830 us; error 28 us; skew 500 ppm
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:33.273258 25516 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:33.273690 25516 webserver.cc:460] Webserver started at http://127.16.122.190:46281/ using document root <none> and password file <none>
I20260923 23:10:33.273897 25516 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:33.273959 25516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:33.275209 25516 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:33.275759 25531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.275898 25516 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.275977 25516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "c63b7c663e674c73b99b8c395e13599d"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "63c694848630e5bba1ebcc889dbe01e6"
server_key_iv: "676deb24dd141f3957b0a9413e0eac5d"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.276296 25516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:33.295002 25516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:33.296832 25516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:33.296969 25516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:33.300513 25516 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:35905
I20260923 23:10:33.300554 25583 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:35905 every 8 connection(s)
I20260923 23:10:33.300902 25516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:33.302274 25584 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:10:33.302906 25584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Bootstrap starting.
I20260923 23:10:33.307507 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25516
I20260923 23:10:33.307857 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:40235
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:35905
--webserver_port=40877
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:10:33.311321 25584 log.cc:824] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:33.312945 25584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:10:33.313201 25584 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Bootstrap complete.
I20260923 23:10:33.315274 25584 raft_consensus.cc:348] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:33.315500 25584 raft_consensus.cc:729] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: c63b7c663e674c73b99b8c395e13599d, State: Initialized, Role: FOLLOWER
I20260923 23:10:33.315670 25584 consensus_queue.cc:260] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:33.315747 25584 raft_consensus.cc:388] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:33.315793 25584 raft_consensus.cc:482] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:33.315840 25584 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:10:33.316781 25584 raft_consensus.cc:504] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:33.316874 25584 leader_election.cc:302] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c63b7c663e674c73b99b8c395e13599d; no voters:
I20260923 23:10:33.317029 25584 leader_election.cc:288] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:10:33.317112 25588 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:10:33.317301 25584 sys_catalog.cc:564] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:33.317313 25588 raft_consensus.cc:686] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [term 2 LEADER]: Becoming Leader. State: Replica: c63b7c663e674c73b99b8c395e13599d, State: Running, Role: LEADER
I20260923 23:10:33.317433 25588 consensus_queue.cc:237] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } }
I20260923 23:10:33.317806 25589 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "c63b7c663e674c73b99b8c395e13599d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } } }
I20260923 23:10:33.317966 25589 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:33.317819 25590 sys_catalog.cc:455] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c63b7c663e674c73b99b8c395e13599d. Latest consensus state: current_term: 2 leader_uuid: "c63b7c663e674c73b99b8c395e13599d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c63b7c663e674c73b99b8c395e13599d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 35905 } } }
I20260923 23:10:33.318102 25590 sys_catalog.cc:458] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:33.318318 25598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:33.318814 25598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:33.318933 25598 catalog_manager.cc:1259] Loaded cluster ID: 5b95f26033e04a38b060d9f8766e7688
I20260923 23:10:33.318967 25598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:33.319778 25598 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:33.319947 25598 catalog_manager.cc:5879] T 00000000000000000000000000000000 P c63b7c663e674c73b99b8c395e13599d: Loaded TSK: 0
I20260923 23:10:33.320154 25598 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:10:33.401021 25586 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:33.401232 25586 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:33.401255 25586 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:33.402441 25586 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:33.402478 25586 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:33.402503 25586 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:33.403947 25586 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.129.keytab
--principal=oryx/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:40235
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=40877
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25586
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:33.404166 25586 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:33.404371 25586 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:33.409072 25611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.409101 25613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.409348 25586 server_base.cc:1034] running on GCE node
W20260923 23:10:33.409452 25610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.409663 25586 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:33.409965 25586 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:33.411104 25586 hybrid_clock.cc:648] HybridClock initialized: now 1790205033411092 us; error 28 us; skew 500 ppm
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:33.414155 25586 init.cc:377] Logged in from keytab as oryx/127.16.122.129@KRBTEST.COM (short username oryx)
I20260923 23:10:33.414717 25586 webserver.cc:460] Webserver started at http://127.16.122.129:40877/ using document root <none> and password file <none>
I20260923 23:10:33.414976 25586 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:33.415088 25586 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:33.416872 25586 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:33.417634 25620 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.417809 25586 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:33.417891 25586 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "b228f2d8984948bc963073a6b13ac480"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "75b97c11669f13b4f9ea22dc2427718b"
server_key_iv: "f53167d8171f87b093b7f165dbb6aca4"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.418215 25586 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:33.439952 25586 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:33.442116 25586 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:33.442263 25586 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:33.442485 25586 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:33.442781 25586 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:33.442821 25586 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.442849 25586 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:33.442867 25586 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.448514 25586 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:40235
I20260923 23:10:33.448567 25733 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:40235 every 8 connection(s)
I20260923 23:10:33.448889 25586 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:33.453166 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25586
I20260923 23:10:33.453456 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:38905
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:35905
--webserver_port=37927
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.457791 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.449715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:50123)
0923 23:10:33.450009 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.450065 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.450691 (+ 626us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.451152 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.451156 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.451856 (+ 700us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.452124 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.452131 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.453199 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.453200 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.454122 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.454125 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.454800 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.454801 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.454932 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.455545 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.455556 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.457589 (+ 2033us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":248,"thread_start_us":109,"threads_started":1}
W20260923 23:10:33.458037 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.449744 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50123 (local address 127.16.122.190:35905)
0923 23:10:33.449912 (+ 168us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.449921 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.450203 (+ 282us) server_negotiation.cc:407] Connection header received
0923 23:10:33.450898 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.450904 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.450948 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.451114 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.452231 (+ 1117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.453097 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.454203 (+ 1106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.454602 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.454702 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.457717 (+ 3015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:50123: BlockingRecv error: recv got EOF from 127.16.122.129:50123 (error 108)
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":74,"threads_started":1}
W20260923 23:10:33.458096 25734 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:35905 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.122.190:35905: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.462980 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.458302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:60845)
0923 23:10:33.459091 (+ 789us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.459100 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.459146 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.459261 (+ 115us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.459262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.459375 (+ 113us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.459416 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.459418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.459991 (+ 573us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.459992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.460473 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.460474 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.460796 (+ 322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.460796 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.460854 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.461272 (+ 418us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.461275 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.462919 (+ 1644us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":774}
W20260923 23:10:33.463084 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.458330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:60845 (local address 127.16.122.190:35905)
0923 23:10:33.458450 (+ 120us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.458452 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.459142 (+ 690us) server_negotiation.cc:407] Connection header received
0923 23:10:33.459200 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.459201 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.459214 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.459255 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.459476 (+ 221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.459929 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.460539 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.460738 (+ 199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.460790 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.463003 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:60845: BlockingRecv error: recv got EOF from 127.16.122.129:60845 (error 108)
Metrics: {"server-negotiator.queue_time_us":105,"thread_start_us":72,"threads_started":1}
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.466940 25737 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.463245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:38499)
0923 23:10:33.463353 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.463362 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.463401 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.463514 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.463515 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.463624 (+ 109us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.463648 (+ 24us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.463649 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.464200 (+ 551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.464201 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.464683 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.464684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.464973 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.464974 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.465020 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.465400 (+ 380us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.465404 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.466878 (+ 1474us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":69,"threads_started":1}
W20260923 23:10:33.467002 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.463297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:38499 (local address 127.16.122.190:35905)
0923 23:10:33.463328 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.463330 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.463400 (+ 70us) server_negotiation.cc:407] Connection header received
0923 23:10:33.463452 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.463453 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.463462 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.463510 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.463706 (+ 196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.464139 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.464750 (+ 611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.464920 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.464970 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.466956 (+ 1986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:38499: BlockingRecv error: recv got EOF from 127.16.122.129:38499 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 23:10:33.467360 25734 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:10:33.534956 25738 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:33.535179 25738 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:33.535203 25738 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:33.536310 25738 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:33.536340 25738 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:33.536365 25738 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:33.537693 25738 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.130.keytab
--principal=oryx/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:38905
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=37927
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25738
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:33.537874 25738 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:33.538040 25738 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:33.542639 25745 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.542686 25746 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.542663 25738 server_base.cc:1034] running on GCE node
W20260923 23:10:33.542647 25748 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.543134 25738 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:33.543376 25738 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:33.544512 25738 hybrid_clock.cc:648] HybridClock initialized: now 1790205033544500 us; error 28 us; skew 500 ppm
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:33.546803 25738 init.cc:377] Logged in from keytab as oryx/127.16.122.130@KRBTEST.COM (short username oryx)
I20260923 23:10:33.547302 25738 webserver.cc:460] Webserver started at http://127.16.122.130:37927/ using document root <none> and password file <none>
I20260923 23:10:33.547506 25738 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:33.547569 25738 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:33.548888 25738 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:33.549379 25755 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.549525 25738 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:33.549604 25738 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "c1364130c0434c4da9f95bfdc05b5d4e"
format_stamp: "Formatted at 2026-09-23 23:10:32 on dist-test-slave-95hh"
server_key: "0968d3dc13890a5c86e8e10803ebbc5b"
server_key_iv: "8a6743e763d044d09c1b8529413b1fae"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.549928 25738 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:33.562352 25738 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:33.563990 25738 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:33.564126 25738 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:33.564320 25738 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:33.564595 25738 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:33.564628 25738 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.564656 25738 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:33.564677 25738 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.570001 25738 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:38905
I20260923 23:10:33.570056 25868 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:38905 every 8 connection(s)
I20260923 23:10:33.570369 25738 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:10:33.577066 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.570990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37395 (local address 127.16.122.190:35905)
0923 23:10:33.571051 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.571053 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.571274 (+ 221us) server_negotiation.cc:407] Connection header received
0923 23:10:33.571775 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.571776 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.571795 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.571847 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.572759 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.573263 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.574240 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.574424 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.574479 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.577010 (+ 2531us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:37395: BlockingRecv error: recv got EOF from 127.16.122.130:37395 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:10:33.577113 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.570987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:37395)
0923 23:10:33.571158 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.571192 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.571676 (+ 484us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.571920 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.571924 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.572468 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.572665 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.572669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.573344 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.573345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.574157 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.574159 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.574490 (+ 331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.574491 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.574618 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.575045 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.575055 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.576921 (+ 1866us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":98,"threads_started":1}
W20260923 23:10:33.577312 25869 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:35905 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.122.190:35905: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
I20260923 23:10:33.578802 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25738
I20260923 23:10:33.579089 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:44467
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:35905
--webserver_port=38125
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.581758 25871 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.577518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:41817)
0923 23:10:33.577560 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.577570 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.577614 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.577739 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.577739 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.577841 (+ 102us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.577885 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.577886 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.578458 (+ 572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.578458 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.579069 (+ 611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.579071 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.579444 (+ 373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.579445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.579492 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.579885 (+ 393us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.579889 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.581692 (+ 1803us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 23:10:33.581836 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.577568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:41817 (local address 127.16.122.190:35905)
0923 23:10:33.577617 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.577619 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.577627 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:33.577679 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.577680 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.577687 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.577728 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.577932 (+ 204us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.578400 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.579145 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.579378 (+ 233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.579445 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.581782 (+ 2337us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:41817: BlockingRecv error: recv got EOF from 127.16.122.130:41817 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.586383 25873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.581986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:38739)
0923 23:10:33.582110 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.582120 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.582171 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.582289 (+ 118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.582289 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.582410 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.582456 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.582457 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.583340 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.583341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.583883 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.583885 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.584319 (+ 434us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.584319 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.584358 (+ 39us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.584778 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.584782 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.586327 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":77,"threads_started":1}
W20260923 23:10:33.586483 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.582033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:38739 (local address 127.16.122.190:35905)
0923 23:10:33.582084 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.582086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.582177 (+ 91us) server_negotiation.cc:407] Connection header received
0923 23:10:33.582222 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.582223 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.582232 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.582281 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.582526 (+ 245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.583270 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.583955 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.584242 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.584312 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.586423 (+ 2111us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:38739: BlockingRecv error: recv got EOF from 127.16.122.130:38739 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
W20260923 23:10:33.586493 25869 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:10:33.658387 25872 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:33.658591 25872 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:33.658620 25872 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:33.659752 25872 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:33.659791 25872 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:33.659824 25872 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:33.661182 25872 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/krb5kdc/oryx_127.16.122.131.keytab
--principal=oryx/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:44467
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=38125
--tserver_master_addrs=127.16.122.190:35905
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.25872
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:33.661397 25872 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:33.661600 25872 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:33.666154 25879 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:33.666153 25881 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.666250 25872 server_base.cc:1034] running on GCE node
W20260923 23:10:33.666193 25878 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:33.666512 25872 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:33.666725 25872 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:33.667856 25872 hybrid_clock.cc:648] HybridClock initialized: now 1790205033667843 us; error 32 us; skew 500 ppm
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205033, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:33.670082 25872 init.cc:377] Logged in from keytab as oryx/127.16.122.131@KRBTEST.COM (short username oryx)
I20260923 23:10:33.670460 25872 webserver.cc:460] Webserver started at http://127.16.122.131:38125/ using document root <none> and password file <none>
I20260923 23:10:33.670641 25872 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:33.670701 25872 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:33.671957 25872 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:33.672480 25888 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.672627 25872 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.672703 25872 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "08d7e999df0b491a9cdec853e3994b18"
format_stamp: "Formatted at 2026-09-23 23:10:33 on dist-test-slave-95hh"
server_key: "9850753c9da724062ed6316334b6115b"
server_key_iv: "3d6e7c95f923897132eefdd5583f5a4e"
server_key_version: "encryptionkey@0"
I20260923 23:10:33.672995 25872 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:33.696795 25872 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:33.698935 25872 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:33.699103 25872 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:33.699337 25872 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:33.699642 25872 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:33.699678 25872 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.699707 25872 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:33.699730 25872 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:33.704872 25872 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:44467
I20260923 23:10:33.704936 26001 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:44467 every 8 connection(s)
I20260923 23:10:33.705245 25872 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
W20260923 23:10:33.711884 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.705899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36205 (local address 127.16.122.190:35905)
0923 23:10:33.705958 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.705961 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.706177 (+ 216us) server_negotiation.cc:407] Connection header received
0923 23:10:33.706665 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.706666 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.706675 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.706715 (+ 40us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.707721 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.708191 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.709090 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.709264 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.709305 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.711838 (+ 2533us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:36205: BlockingRecv error: recv got EOF from 127.16.122.131:36205 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:10:33.711903 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.705892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:36205)
0923 23:10:33.706047 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.706097 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.706556 (+ 459us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.706774 (+ 218us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.706777 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.707396 (+ 619us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.707643 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.707648 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.708255 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.708256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.709031 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.709033 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.709315 (+ 282us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.709316 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.709406 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.709809 (+ 403us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.709817 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.711767 (+ 1950us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":70,"threads_started":1}
W20260923 23:10:33.712182 26002 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:35905 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.122.190:35905: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
W20260923 23:10:33.712503 25583 acceptor_pool.cc:285] unable to collect diagnostics on RPC socket 127.16.122.190:35905: Invalid argument: socket address is ambiguous
I20260923 23:10:33.714306 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 25872
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.716537 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.712403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:60367)
0923 23:10:33.712442 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.712451 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.712483 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.712716 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.712717 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.712813 (+ 96us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.712853 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.712854 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.713383 (+ 529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.713384 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.713858 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.713859 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.714498 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.714499 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.714546 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.714927 (+ 381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.714931 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.716476 (+ 1545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":23}
W20260923 23:10:33.716595 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.712601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:60367 (local address 127.16.122.190:35905)
0923 23:10:33.712633 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.712635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.712642 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:10:33.712658 (+ 16us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.712659 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.712665 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.712704 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.712898 (+ 194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.713337 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.713944 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.714125 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.714186 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.716547 (+ 2361us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:60367: BlockingRecv error: recv got EOF from 127.16.122.131:60367 (error 108)
Metrics: {"server-negotiator.queue_time_us":15}
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:33 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:33.720769 26004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.716949 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:43073)
0923 23:10:33.716987 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:33.716996 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:33.717030 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:33.717127 (+ 97us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:33.717127 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:33.717243 (+ 116us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:33.717265 (+ 22us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.717266 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.717940 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.717941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.718494 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:33.718496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.718880 (+ 384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.718881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:33.718931 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.719285 (+ 354us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:33.719290 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:33.720709 (+ 1419us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:10:33.720810 25736 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:33.716999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:43073 (local address 127.16.122.190:35905)
0923 23:10:33.717038 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.717040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.717048 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:33.717067 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.717068 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.717075 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.717119 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.717320 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.717876 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.718562 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.718816 (+ 254us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.718869 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.720771 (+ 1902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:43073: BlockingRecv error: recv got EOF from 127.16.122.131:43073 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:10:33.720988 26002 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:10:33.721477 25739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:33.714819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57740 (local address 127.16.122.190:35905)
0923 23:10:33.714876 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:33.714878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:33.714886 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:33.714949 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:33.714950 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:33.714961 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:33.715022 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:33.715674 (+ 652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.716270 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.717110 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:33.717326 (+ 216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:33.717404 (+ 78us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:33.718542 (+ 1138us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:33.718547 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:33.718548 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:33.718559 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:33.720058 (+ 1499us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:33.720353 (+ 295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:33.720357 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:33.720357 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:33.720474 (+ 117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:33.720661 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:33.720662 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:33.720662 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:33.721010 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:33.721143 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:33.721337 (+ 194us) server_negotiation.cc:299] Negotiation successful
0923 23:10:33.721422 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:10:34.301944 25579 messenger.cc:188] Invalid argument: socket address is ambiguous: no data on pending connections for acceptor pool at 127.16.122.190:35905
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:34.472088 26006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:34.467706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:49257)
0923 23:10:34.467822 (+ 116us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:34.467832 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:34.467880 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:34.468016 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:34.468016 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:34.468167 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:34.468216 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.468218 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.468836 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.468837 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.469375 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.469377 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.469745 (+ 368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.469747 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.469803 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.470254 (+ 451us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:34.470257 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:34.472018 (+ 1761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
W20260923 23:10:34.472209 26007 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:34.467760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:49257 (local address 127.16.122.190:35905)
0923 23:10:34.467892 (+ 132us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:34.467894 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:34.467905 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:10:34.467927 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:34.467927 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:34.467938 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:34.467999 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:34.468272 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.468770 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.469443 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.469653 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.469716 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.472126 (+ 2410us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:49257: BlockingRecv error: recv got EOF from 127.16.122.129:49257 (error 108)
Metrics: {"server-negotiator.queue_time_us":103,"thread_start_us":61,"threads_started":1}
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:34.591276 26008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:34.586872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:52335)
0923 23:10:34.586996 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:34.587008 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:34.587088 (+ 80us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:34.587192 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:34.587193 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:34.587383 (+ 190us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:34.587437 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.587439 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.588088 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.588088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.588630 (+ 542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.588631 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.588934 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.588935 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.588995 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.589410 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:34.589415 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:34.591203 (+ 1788us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
W20260923 23:10:34.591351 26007 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:34.586947 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:52335 (local address 127.16.122.190:35905)
0923 23:10:34.587013 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:34.587015 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:34.587093 (+ 78us) server_negotiation.cc:407] Connection header received
0923 23:10:34.587125 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:34.587126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:34.587137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:34.587181 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:34.587529 (+ 348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.588018 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.588697 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.588876 (+ 179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.588924 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.591295 (+ 2371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:52335: BlockingRecv error: recv got EOF from 127.16.122.130:52335 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:34 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:34.725646 26009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:34.721347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:36237)
0923 23:10:34.721469 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:34.721481 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:34.721555 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:34.721715 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:34.721716 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:34.721883 (+ 167us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:34.721933 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.721935 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.722566 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.722567 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.723156 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:34.723158 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.723476 (+ 318us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.723477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:34.723531 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.723940 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:34.723944 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:34.725557 (+ 1613us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":65,"threads_started":1}
W20260923 23:10:34.725754 26007 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:34.721406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36237 (local address 127.16.122.190:35905)
0923 23:10:34.721474 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:34.721476 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:34.721547 (+ 71us) server_negotiation.cc:407] Connection header received
0923 23:10:34.721645 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:34.721646 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:34.721657 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:34.721703 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:34.722006 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.722495 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.723235 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:34.723419 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:34.723473 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:34.725700 (+ 2227us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:36237: BlockingRecv error: recv got EOF from 127.16.122.131:36237 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:35.477035 26010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:35.472594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:44259)
0923 23:10:35.472756 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:35.472765 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:35.472817 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:35.472950 (+ 133us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:35.472951 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:35.473098 (+ 147us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:35.473147 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.473149 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.473782 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.473783 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.474313 (+ 530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.474315 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.474653 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.474653 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.474690 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.475189 (+ 499us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:35.475193 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:35.476955 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":144,"thread_start_us":65,"threads_started":1}
W20260923 23:10:35.477118 26011 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:35.472677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44259 (local address 127.16.122.190:35905)
0923 23:10:35.472788 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:35.472790 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:35.472802 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:35.472877 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:35.472878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:35.472887 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:35.472943 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:35.473197 (+ 254us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.473709 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.474386 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.474580 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.474654 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.477048 (+ 2394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:44259: BlockingRecv error: recv got EOF from 127.16.122.129:44259 (error 108)
Metrics: {"server-negotiator.queue_time_us":91,"thread_start_us":61,"threads_started":1}
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:35.596629 26012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:35.591879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:44631)
0923 23:10:35.592024 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:35.592037 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:35.592120 (+ 83us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:35.592261 (+ 141us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:35.592262 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:35.592431 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:35.592494 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.592496 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.593125 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.593126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.593679 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.593682 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.594068 (+ 386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.594069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.594123 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.594611 (+ 488us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:35.594615 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:35.596546 (+ 1931us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":77,"threads_started":1}
W20260923 23:10:35.596704 26011 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:35.591982 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:44631 (local address 127.16.122.190:35905)
0923 23:10:35.592062 (+ 80us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:35.592064 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:35.592114 (+ 50us) server_negotiation.cc:407] Connection header received
0923 23:10:35.592175 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:35.592176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:35.592186 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:35.592246 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:35.592550 (+ 304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.593051 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.593783 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.593977 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.594040 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.596653 (+ 2613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:44631: BlockingRecv error: recv got EOF from 127.16.122.130:44631 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:35 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:35.730473 26013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:35.726134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:47479)
0923 23:10:35.726252 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:35.726262 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:35.726315 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:35.726422 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:35.726422 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:35.726584 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:35.726630 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.726632 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.727281 (+ 649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.727282 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.727859 (+ 577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:35.727862 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.728198 (+ 336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.728199 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:35.728240 (+ 41us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.728664 (+ 424us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:35.728668 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:35.730415 (+ 1747us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":75,"threads_started":1}
W20260923 23:10:35.730571 26011 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:35.726257 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:47479 (local address 127.16.122.190:35905)
0923 23:10:35.726315 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:35.726317 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:35.726347 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:10:35.726368 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:35.726368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:35.726380 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:35.726416 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:35.726692 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.727199 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.727944 (+ 745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:35.728138 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:35.728191 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:35.730509 (+ 2318us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:47479: BlockingRecv error: recv got EOF from 127.16.122.131:47479 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:36.481904 26014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:36.477455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:35187)
0923 23:10:36.477599 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:36.477609 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:36.477663 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:36.477805 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:36.477806 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:36.477963 (+ 157us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:36.478012 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.478013 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.478843 (+ 830us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.478844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.479428 (+ 584us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.479431 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.479776 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.479776 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.479823 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.480262 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:36.480266 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:36.481846 (+ 1580us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":93,"threads_started":1}
W20260923 23:10:36.481998 26015 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:36.477526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:35187 (local address 127.16.122.190:35905)
0923 23:10:36.477633 (+ 107us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:36.477635 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:36.477648 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:10:36.477720 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:36.477720 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:36.477732 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:36.477788 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:36.478070 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.478778 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.479492 (+ 714us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.479705 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.479764 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.481933 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:35187: BlockingRecv error: recv got EOF from 127.16.122.129:35187 (error 108)
Metrics: {"server-negotiator.queue_time_us":76,"thread_start_us":46,"threads_started":1}
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:36.601233 26016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:36.597088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:37883)
0923 23:10:36.597199 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:36.597208 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:36.597254 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:36.597374 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:36.597375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:36.597525 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:36.597573 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.597574 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.598233 (+ 659us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.598234 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.598747 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.598748 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.599135 (+ 387us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.599135 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.599178 (+ 43us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.599608 (+ 430us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:36.599613 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:36.601174 (+ 1561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":64,"threads_started":1}
W20260923 23:10:36.601425 26015 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:36.597171 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37883 (local address 127.16.122.190:35905)
0923 23:10:36.597243 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:36.597245 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:36.597258 (+ 13us) server_negotiation.cc:407] Connection header received
0923 23:10:36.597308 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:36.597309 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:36.597320 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:36.597366 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:36.597661 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.598172 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.598833 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.599025 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.599134 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.601367 (+ 2233us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:37883: BlockingRecv error: recv got EOF from 127.16.122.130:37883 (error 108)
Metrics: {"server-negotiator.queue_time_us":46}
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:36 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:36.738281 26017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:36.730940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:55115)
0923 23:10:36.731099 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:36.731112 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:36.731187 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:36.731317 (+ 130us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:36.731318 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:36.731477 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:36.731523 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.731524 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.732124 (+ 600us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.732124 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.732629 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:36.732630 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.732921 (+ 291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.732922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:36.732957 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.736597 (+ 3640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:36.736601 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:36.738221 (+ 1620us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":138,"thread_start_us":59,"threads_started":1}
W20260923 23:10:36.738374 26015 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:36.731065 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:55115 (local address 127.16.122.190:35905)
0923 23:10:36.731132 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:36.731134 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:36.731166 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:10:36.731247 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:36.731248 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:36.731261 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:36.731303 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:36.731568 (+ 265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.732067 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.732693 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:36.732871 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:36.732920 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:36.738310 (+ 5390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:55115: BlockingRecv error: recv got EOF from 127.16.122.131:55115 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:37.486750 26018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:37.482311 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:58625)
0923 23:10:37.482429 (+ 118us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:37.482442 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:37.482499 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:37.482606 (+ 107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:37.482607 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:37.482756 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:37.482805 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.482807 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.483474 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.483475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.484010 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.484012 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.484386 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.484386 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.484422 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.484866 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:37.484870 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:37.486671 (+ 1801us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":56,"threads_started":1}
W20260923 23:10:37.486867 26019 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:37.482371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:58625 (local address 127.16.122.190:35905)
0923 23:10:37.482497 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:37.482499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:37.482508 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:37.482532 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:37.482532 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:37.482541 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:37.482596 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:37.482887 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.483415 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.484102 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.484322 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.484384 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.486798 (+ 2414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:58625: BlockingRecv error: recv got EOF from 127.16.122.129:58625 (error 108)
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":64,"threads_started":1}
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:37.609473 26020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:37.601712 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:59533)
0923 23:10:37.601868 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:37.601880 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:37.601955 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:37.602078 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:37.602079 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:37.602257 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:37.602308 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.602310 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.602929 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.602930 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.603502 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.603504 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.603828 (+ 324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.603829 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.603881 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.607667 (+ 3786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:37.607672 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:37.609391 (+ 1719us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":64,"threads_started":1}
W20260923 23:10:37.609526 26019 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:37.601811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:59533 (local address 127.16.122.190:35905)
0923 23:10:37.601882 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:37.601884 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:37.601945 (+ 61us) server_negotiation.cc:407] Connection header received
0923 23:10:37.602003 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:37.602004 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:37.602014 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:37.602059 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:37.602361 (+ 302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.602846 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.603569 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.603746 (+ 177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.603794 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.609477 (+ 5683us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:59533: BlockingRecv error: recv got EOF from 127.16.122.130:59533 (error 108)
Metrics: {"server-negotiator.queue_time_us":34}
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:37 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:37.742779 26021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:37.738698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:39965)
0923 23:10:37.738822 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:37.738833 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:37.738894 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:37.739005 (+ 111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:37.739006 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:37.739241 (+ 235us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:37.739292 (+ 51us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.739294 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.739880 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.739881 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.740398 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:37.740399 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.740663 (+ 264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.740664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:37.740699 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.741108 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:37.741113 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:37.742721 (+ 1608us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":61,"threads_started":1}
W20260923 23:10:37.742869 26019 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:37.738782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:39965 (local address 127.16.122.190:35905)
0923 23:10:37.738825 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:37.738827 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:37.738892 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:10:37.738934 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:37.738934 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:37.738947 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:37.738990 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:37.739351 (+ 361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.739825 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.740451 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:37.740617 (+ 166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:37.740658 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:37.742810 (+ 2152us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:39965: BlockingRecv error: recv got EOF from 127.16.122.131:39965 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:38.491820 26022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:38.487335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:54179)
0923 23:10:38.487468 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:38.487477 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:38.487524 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:38.487655 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:38.487655 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:38.487801 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:38.487847 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.487849 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.488435 (+ 586us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.488436 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.488938 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.488939 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.489239 (+ 300us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.489240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.489275 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.489801 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:38.489805 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:38.491744 (+ 1939us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":68,"threads_started":1}
W20260923 23:10:38.491971 26023 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:38.487399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:54179 (local address 127.16.122.190:35905)
0923 23:10:38.487499 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:38.487501 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:38.487513 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:38.487571 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:38.487572 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:38.487582 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:38.487643 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:38.487895 (+ 252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.488384 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.488993 (+ 609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.489189 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.489237 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.491899 (+ 2662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:54179: BlockingRecv error: recv got EOF from 127.16.122.129:54179 (error 108)
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":51,"threads_started":1}
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:38.614073 26024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:38.609940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:49895)
0923 23:10:38.610090 (+ 150us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:38.610101 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:38.610165 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:38.610287 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:38.610288 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:38.610434 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:38.610477 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.610479 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.611139 (+ 660us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.611140 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.611665 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.611667 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.612005 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.612006 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.612040 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.612435 (+ 395us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:38.612439 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:38.614010 (+ 1571us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":70,"threads_started":1}
W20260923 23:10:38.614143 26023 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:38.610058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:49895 (local address 127.16.122.190:35905)
0923 23:10:38.610124 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:38.610126 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:38.610140 (+ 14us) server_negotiation.cc:407] Connection header received
0923 23:10:38.610224 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:38.610224 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:38.610235 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:38.610273 (+ 38us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:38.610543 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.611074 (+ 531us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.611751 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.611952 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.612005 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.614086 (+ 2081us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:49895: BlockingRecv error: recv got EOF from 127.16.122.130:49895 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:38 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:38.747429 26025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:38.743276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:51371)
0923 23:10:38.743411 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:38.743422 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:38.743480 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:38.743602 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:38.743603 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:38.743755 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:38.743800 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.743802 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.744395 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.744396 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.744904 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:38.744905 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.745212 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.745213 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:38.745246 (+ 33us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.745635 (+ 389us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:38.745639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:38.747348 (+ 1709us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":56,"threads_started":1}
W20260923 23:10:38.747479 26023 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:38.743350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:51371 (local address 127.16.122.190:35905)
0923 23:10:38.743391 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:38.743393 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:38.743474 (+ 81us) server_negotiation.cc:407] Connection header received
0923 23:10:38.743522 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:38.743523 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:38.743535 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:38.743586 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:38.743848 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.744336 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.744968 (+ 632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:38.745151 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:38.745211 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:38.747431 (+ 2220us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:51371: BlockingRecv error: recv got EOF from 127.16.122.131:51371 (error 108)
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:39.497512 26026 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:39.492535 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:50401)
0923 23:10:39.492665 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:39.492675 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:39.492728 (+ 53us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:39.492888 (+ 160us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:39.492890 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:39.493046 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:39.493106 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.493108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.493870 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.493872 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.494583 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.494585 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.495063 (+ 478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.495064 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.495115 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.495582 (+ 467us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:39.495587 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:39.497435 (+ 1848us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
W20260923 23:10:39.497596 26027 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:39.492600 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50401 (local address 127.16.122.190:35905)
0923 23:10:39.492719 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:39.492721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:39.492731 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:39.492771 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:39.492771 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:39.492781 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:39.492852 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:39.493172 (+ 320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.493784 (+ 612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.494665 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.494978 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.495069 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.497534 (+ 2465us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:50401: BlockingRecv error: recv got EOF from 127.16.122.129:50401 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":62,"threads_started":1}
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:39.618639 26028 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:39.614521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:46523)
0923 23:10:39.614632 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:39.614641 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:39.614692 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:39.614802 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:39.614803 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:39.614946 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:39.614993 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.614994 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.615649 (+ 655us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.615649 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.616155 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.616157 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.616460 (+ 303us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.616460 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.616496 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.616911 (+ 415us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:39.616915 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:39.618583 (+ 1668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":53,"threads_started":1}
W20260923 23:10:39.618727 26027 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:39.614582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:46523 (local address 127.16.122.190:35905)
0923 23:10:39.614634 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:39.614636 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:39.614681 (+ 45us) server_negotiation.cc:407] Connection header received
0923 23:10:39.614734 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:39.614735 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:39.614745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:39.614795 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:39.615059 (+ 264us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.615583 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.616222 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.616406 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.616458 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.618669 (+ 2211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:46523: BlockingRecv error: recv got EOF from 127.16.122.130:46523 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:39 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:39.752103 26029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:39.747866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:34941)
0923 23:10:39.747979 (+ 113us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:39.747993 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:39.748067 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:39.748206 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:39.748207 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:39.748371 (+ 164us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:39.748420 (+ 49us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.748422 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.749051 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.749052 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.749585 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:39.749587 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.749868 (+ 281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.749869 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:39.749906 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.750361 (+ 455us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:39.750365 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:39.752030 (+ 1665us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":51,"threads_started":1}
W20260923 23:10:39.752162 26027 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:39.747966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:34941 (local address 127.16.122.190:35905)
0923 23:10:39.748056 (+ 90us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:39.748058 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:39.748067 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:39.748125 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:39.748126 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:39.748137 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:39.748182 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:39.748477 (+ 295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.748962 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.749648 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:39.749819 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:39.749865 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:39.752113 (+ 2248us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:34941: BlockingRecv error: recv got EOF from 127.16.122.131:34941 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:40.502776 26030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:40.497926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:40713)
0923 23:10:40.498165 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:40.498178 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:40.498262 (+ 84us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:40.498473 (+ 211us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:40.498474 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:40.498673 (+ 199us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:40.498730 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.498732 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.499624 (+ 892us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.499625 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.500148 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.500150 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.500455 (+ 305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.500455 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.500491 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.500947 (+ 456us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:40.500952 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:40.502660 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":215,"thread_start_us":64,"threads_started":1}
W20260923 23:10:40.502799 26031 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:40.498046 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:40713 (local address 127.16.122.190:35905)
0923 23:10:40.498155 (+ 109us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:40.498157 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:40.498259 (+ 102us) server_negotiation.cc:407] Connection header received
0923 23:10:40.498331 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:40.498332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:40.498349 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:40.498418 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:40.498812 (+ 394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.499552 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.500214 (+ 662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.500407 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.500449 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.502747 (+ 2298us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:40713: BlockingRecv error: recv got EOF from 127.16.122.129:40713 (error 108)
Metrics: {"server-negotiator.queue_time_us":85,"thread_start_us":60,"threads_started":1}
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:40.623687 26032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:40.619127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:41443)
0923 23:10:40.619259 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:40.619271 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:40.619346 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:40.619477 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:40.619478 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:40.619654 (+ 176us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:40.619709 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.619711 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.620490 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.620492 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.621144 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.621148 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.621487 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.621488 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.621544 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.621943 (+ 399us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:40.621947 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:40.623629 (+ 1682us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":64,"threads_started":1}
W20260923 23:10:40.623782 26031 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:40.619181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:41443 (local address 127.16.122.190:35905)
0923 23:10:40.619247 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:40.619249 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:40.619332 (+ 83us) server_negotiation.cc:407] Connection header received
0923 23:10:40.619406 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:40.619408 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:40.619420 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:40.619468 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:40.619779 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.620415 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.621232 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.621418 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.621480 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.623722 (+ 2242us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:41443: BlockingRecv error: recv got EOF from 127.16.122.130:41443 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:40 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:40.756776 26033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:40.752500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:33973)
0923 23:10:40.752648 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:40.752660 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:40.752739 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:40.752881 (+ 142us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:40.752882 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:40.753053 (+ 171us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:40.753099 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.753101 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.753817 (+ 716us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.753818 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.754332 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:40.754334 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.754653 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.754654 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:40.754714 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.755154 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:40.755158 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:40.756717 (+ 1559us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":82,"threads_started":1}
W20260923 23:10:40.756855 26031 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:40.752569 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:33973 (local address 127.16.122.190:35905)
0923 23:10:40.752615 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:40.752618 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:40.752727 (+ 109us) server_negotiation.cc:407] Connection header received
0923 23:10:40.752799 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:40.752800 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:40.752813 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:40.752856 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:40.753170 (+ 314us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.753743 (+ 573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.754404 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:40.754596 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:40.754649 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:40.756798 (+ 2149us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:33973: BlockingRecv error: recv got EOF from 127.16.122.131:33973 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:41.507294 26034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:41.503210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:45135)
0923 23:10:41.503333 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:41.503347 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:41.503402 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:41.503534 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:41.503534 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:41.503682 (+ 148us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:41.503738 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.503740 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.504331 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.504332 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.504865 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.504867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.505155 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.505156 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.505192 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.505594 (+ 402us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:41.505597 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:41.507235 (+ 1638us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
W20260923 23:10:41.507396 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:41.503317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:45135 (local address 127.16.122.190:35905)
0923 23:10:41.503423 (+ 106us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:41.503425 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:41.503435 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:41.503457 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:41.503457 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:41.503467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:41.503522 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:41.503793 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.504277 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.504925 (+ 648us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.505108 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.505151 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.507320 (+ 2169us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:45135: BlockingRecv error: recv got EOF from 127.16.122.129:45135 (error 108)
Metrics: {"server-negotiator.queue_time_us":86,"thread_start_us":45,"threads_started":1}
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:41.628572 26036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:41.624134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:38499)
0923 23:10:41.624261 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:41.624271 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:41.624340 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:41.624467 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:41.624468 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:41.624620 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:41.624668 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.624669 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.625428 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.625429 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.625946 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.625947 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.626239 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.626240 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.626276 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.626716 (+ 440us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:41.626721 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:41.628496 (+ 1775us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":62,"threads_started":1}
W20260923 23:10:41.628646 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:41.624197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:38499 (local address 127.16.122.190:35905)
0923 23:10:41.624278 (+ 81us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:41.624280 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:41.624336 (+ 56us) server_negotiation.cc:407] Connection header received
0923 23:10:41.624398 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:41.624399 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:41.624410 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:41.624449 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:41.624725 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.625368 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.626006 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.626190 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.626236 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.628595 (+ 2359us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:38499: BlockingRecv error: recv got EOF from 127.16.122.130:38499 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:41 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:41.761330 26037 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:41.757205 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:37315)
0923 23:10:41.757326 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:41.757336 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:41.757398 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:41.757508 (+ 110us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:41.757509 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:41.757661 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:41.757705 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.757706 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.758294 (+ 588us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.758294 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.758825 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:41.758826 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.759160 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.759161 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:41.759203 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.759635 (+ 432us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:41.759639 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:41.761255 (+ 1616us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":75,"threads_started":1}
W20260923 23:10:41.761436 26035 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:41.757255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:37315 (local address 127.16.122.190:35905)
0923 23:10:41.757325 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:41.757327 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:41.757384 (+ 57us) server_negotiation.cc:407] Connection header received
0923 23:10:41.757451 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:41.757452 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:41.757463 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:41.757497 (+ 34us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:41.757757 (+ 260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.758241 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.758883 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:41.759112 (+ 229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:41.759163 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:41.761375 (+ 2212us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:37315: BlockingRecv error: recv got EOF from 127.16.122.131:37315 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:42.511986 26038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:42.507719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:58839)
0923 23:10:42.507841 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:42.507852 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:42.507908 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:42.508020 (+ 112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:42.508020 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:42.508169 (+ 149us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:42.508226 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.508228 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.508859 (+ 631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.508860 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.509411 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.509413 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.509693 (+ 280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.509693 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.509735 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.510176 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:42.510180 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:42.511913 (+ 1733us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":63,"threads_started":1}
W20260923 23:10:42.512072 26039 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:42.507802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:58839 (local address 127.16.122.190:35905)
0923 23:10:42.507896 (+ 94us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:42.507898 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:42.507907 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:42.507951 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:42.507952 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:42.507962 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:42.508015 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:42.508299 (+ 284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.508797 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.509465 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.509647 (+ 182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.509690 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.512000 (+ 2310us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:58839: BlockingRecv error: recv got EOF from 127.16.122.129:58839 (error 108)
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:42.639707 26040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:42.629094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:41045)
0923 23:10:42.629256 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:42.629268 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:42.629338 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:42.629457 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:42.629458 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:42.629614 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:42.629661 (+ 47us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.629662 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.630273 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.630274 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.630825 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.630827 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.631147 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.631148 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.631185 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.637889 (+ 6704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:42.637894 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:42.639648 (+ 1754us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":80,"threads_started":1}
W20260923 23:10:42.639799 26039 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:42.629216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:41045 (local address 127.16.122.190:35905)
0923 23:10:42.629264 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:42.629266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:42.629322 (+ 56us) server_negotiation.cc:407] Connection header received
0923 23:10:42.629394 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:42.629395 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:42.629407 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:42.629444 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:42.629712 (+ 268us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.630202 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.630899 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.631099 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.631144 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.639741 (+ 8597us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:41045: BlockingRecv error: recv got EOF from 127.16.122.130:41045 (error 108)
Metrics: {"server-negotiator.queue_time_us":26}
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:42 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:42.766207 26041 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:42.761826 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:36527)
0923 23:10:42.761977 (+ 151us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:42.761990 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:42.762062 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:42.762230 (+ 168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:42.762230 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:42.762408 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:42.762467 (+ 59us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.762470 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.763170 (+ 700us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.763170 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.763683 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:42.763684 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.763973 (+ 289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.763973 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:42.764007 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.764430 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:42.764434 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:42.766150 (+ 1716us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":59,"threads_started":1}
W20260923 23:10:42.766321 26039 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:42.761937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36527 (local address 127.16.122.190:35905)
0923 23:10:42.762008 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:42.762011 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:42.762044 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:10:42.762138 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:42.762140 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:42.762153 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:42.762216 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:42.762541 (+ 325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.763110 (+ 569us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.763749 (+ 639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:42.763924 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:42.763969 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:42.766260 (+ 2291us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:36527: BlockingRecv error: recv got EOF from 127.16.122.131:36527 (error 108)
Metrics: {"server-negotiator.queue_time_us":44}
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:43.516855 26042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:43.512465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:56833)
0923 23:10:43.512600 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:43.512613 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:43.512685 (+ 72us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:43.512820 (+ 135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:43.512821 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:43.512973 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:43.513021 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.513023 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.513635 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.513635 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.514144 (+ 509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.514145 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.514449 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.514449 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.514483 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.514940 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:43.514946 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:43.516778 (+ 1832us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":69,"threads_started":1}
W20260923 23:10:43.516902 26043 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:43.512545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:56833 (local address 127.16.122.190:35905)
0923 23:10:43.512668 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:43.512670 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:43.512679 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:43.512748 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:43.512749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:43.512759 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:43.512807 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:43.513070 (+ 263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.513580 (+ 510us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.514201 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.514397 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.514444 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.516841 (+ 2397us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:56833: BlockingRecv error: recv got EOF from 127.16.122.129:56833 (error 108)
Metrics: {"server-negotiator.queue_time_us":101,"thread_start_us":63,"threads_started":1}
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:43.644882 26044 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:43.640198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:54383)
0923 23:10:43.640325 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:43.640338 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:43.640403 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:43.640524 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:43.640525 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:43.640677 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:43.640749 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.640751 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.641384 (+ 633us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.641385 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.641953 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.641955 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.642333 (+ 378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.642334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.642383 (+ 49us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.642883 (+ 500us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:43.642887 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:43.644813 (+ 1926us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":67,"threads_started":1}
W20260923 23:10:43.644950 26043 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:43.640298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:54383 (local address 127.16.122.190:35905)
0923 23:10:43.640364 (+ 66us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:43.640366 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:43.640389 (+ 23us) server_negotiation.cc:407] Connection header received
0923 23:10:43.640456 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:43.640457 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:43.640467 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:43.640512 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:43.640800 (+ 288us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.641286 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.642053 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.642253 (+ 200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.642316 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.644900 (+ 2584us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:54383: BlockingRecv error: recv got EOF from 127.16.122.130:54383 (error 108)
Metrics: {"server-negotiator.queue_time_us":43}
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:43 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:43.771103 26045 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:43.766715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:40863)
0923 23:10:43.766850 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:43.766864 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:43.766929 (+ 65us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:43.767065 (+ 136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:43.767066 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:43.767239 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:43.767287 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.767289 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.767923 (+ 634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.767923 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.768458 (+ 535us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:43.768459 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.768731 (+ 272us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.768732 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:43.768774 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.769165 (+ 391us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:43.769168 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:43.770985 (+ 1817us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":72,"threads_started":1}
W20260923 23:10:43.771186 26043 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:43.766882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40863 (local address 127.16.122.190:35905)
0923 23:10:43.766943 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:43.766945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:43.766953 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:43.766974 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:43.766975 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:43.766985 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:43.767068 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:43.767353 (+ 285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.767859 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.768514 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:43.768685 (+ 171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:43.768728 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:43.771130 (+ 2402us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:40863: BlockingRecv error: recv got EOF from 127.16.122.131:40863 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:44.521860 26046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:44.517315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:52221)
0923 23:10:44.517449 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:44.517462 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:44.517536 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:44.517661 (+ 125us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:44.517661 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:44.517821 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:44.517865 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.517867 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.518558 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.518559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.519147 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.519150 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.519512 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.519513 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.519566 (+ 53us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.520027 (+ 461us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:44.520032 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:44.521781 (+ 1749us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":59,"threads_started":1}
W20260923 23:10:44.521926 26047 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:44.517367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:52221 (local address 127.16.122.190:35905)
0923 23:10:44.517478 (+ 111us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:44.517480 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:44.517518 (+ 38us) server_negotiation.cc:407] Connection header received
0923 23:10:44.517582 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:44.517583 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:44.517594 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:44.517648 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:44.517924 (+ 276us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.518492 (+ 568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.519218 (+ 726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.519439 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.519505 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.521865 (+ 2360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:52221: BlockingRecv error: recv got EOF from 127.16.122.129:52221 (error 108)
Metrics: {"server-negotiator.queue_time_us":90,"thread_start_us":63,"threads_started":1}
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:44.649564 26048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:44.645322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:42163)
0923 23:10:44.645467 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:44.645477 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:44.645527 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:44.645665 (+ 138us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:44.645665 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:44.645824 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:44.645870 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.645871 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.646529 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.646530 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.647074 (+ 544us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.647076 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.647404 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.647405 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.647447 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.647881 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:44.647885 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:44.649507 (+ 1622us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":77,"threads_started":1}
W20260923 23:10:44.649685 26047 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:44.645411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:42163 (local address 127.16.122.190:35905)
0923 23:10:44.645497 (+ 86us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:44.645499 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:44.645513 (+ 14us) server_negotiation.cc:407] Connection header received
0923 23:10:44.645585 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:44.645586 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:44.645596 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:44.645654 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:44.645972 (+ 318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.646462 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.647152 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.647348 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.647404 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.649617 (+ 2213us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:42163: BlockingRecv error: recv got EOF from 127.16.122.130:42163 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:44 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:44.775969 26049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:44.771614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:47235)
0923 23:10:44.771735 (+ 121us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:44.771744 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:44.771818 (+ 74us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:44.771973 (+ 155us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:44.771974 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:44.772136 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:44.772181 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.772183 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.772801 (+ 618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.772802 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.773352 (+ 550us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:44.773353 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.773645 (+ 292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.773646 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:44.773692 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.774131 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:44.774135 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:44.775897 (+ 1762us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
W20260923 23:10:44.776027 26047 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:44.771684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:47235 (local address 127.16.122.190:35905)
0923 23:10:44.771731 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:44.771733 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:44.771787 (+ 54us) server_negotiation.cc:407] Connection header received
0923 23:10:44.771860 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:44.771861 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:44.771871 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:44.771918 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:44.772247 (+ 329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.772735 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.773413 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:44.773589 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:44.773631 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:44.775978 (+ 2347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:47235: BlockingRecv error: recv got EOF from 127.16.122.131:47235 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:45.526376 26050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:45.522256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:38225)
0923 23:10:45.522380 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:45.522391 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:45.522437 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:45.522580 (+ 143us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:45.522581 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:45.522737 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:45.522783 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.522785 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.523470 (+ 685us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.523471 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.523998 (+ 527us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.524000 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.524304 (+ 304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.524304 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.524341 (+ 37us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.524760 (+ 419us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:45.524765 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:45.526318 (+ 1553us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":76,"threads_started":1}
W20260923 23:10:45.526510 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:45.522327 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:38225 (local address 127.16.122.190:35905)
0923 23:10:45.522463 (+ 136us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:45.522465 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:45.522473 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:45.522493 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:45.522494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:45.522503 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:45.522563 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:45.522841 (+ 278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.523404 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.524058 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.524241 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.524303 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.526428 (+ 2125us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:38225: BlockingRecv error: recv got EOF from 127.16.122.129:38225 (error 108)
Metrics: {"server-negotiator.queue_time_us":118,"thread_start_us":73,"threads_started":1}
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:45.654058 26052 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:45.649953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:37037)
0923 23:10:45.650112 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:45.650122 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:45.650177 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:45.650281 (+ 104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:45.650281 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:45.650442 (+ 161us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:45.650487 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.650489 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.651096 (+ 607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.651097 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.651625 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.651626 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.651903 (+ 277us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.651903 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.651941 (+ 38us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.652362 (+ 421us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:45.652366 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:45.653991 (+ 1625us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":76,"threads_started":1}
W20260923 23:10:45.654134 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:45.650021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37037 (local address 127.16.122.190:35905)
0923 23:10:45.650076 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:45.650079 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:45.650171 (+ 92us) server_negotiation.cc:407] Connection header received
0923 23:10:45.650219 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:45.650220 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:45.650229 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:45.650274 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:45.650536 (+ 262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.651021 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.651682 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.651854 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.651893 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.654075 (+ 2182us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:37037: BlockingRecv error: recv got EOF from 127.16.122.130:37037 (error 108)
Metrics: {"server-negotiator.queue_time_us":28}
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:45 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:45.780990 26053 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:45.776426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:33649)
0923 23:10:45.776585 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:45.776596 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:45.776662 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:45.776781 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:45.776781 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:45.776968 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:45.777020 (+ 52us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.777022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.777758 (+ 736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.777759 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.778291 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:45.778293 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.778581 (+ 288us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.778582 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:45.778629 (+ 47us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.779063 (+ 434us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:45.779068 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:45.780918 (+ 1850us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":139,"thread_start_us":78,"threads_started":1}
W20260923 23:10:45.781049 26051 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:45.776535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:33649 (local address 127.16.122.190:35905)
0923 23:10:45.776583 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:45.776585 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:45.776657 (+ 72us) server_negotiation.cc:407] Connection header received
0923 23:10:45.776705 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:45.776706 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:45.776717 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:45.776768 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:45.777091 (+ 323us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.777695 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.778345 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:45.778526 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:45.778572 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:45.781000 (+ 2428us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:33649: BlockingRecv error: recv got EOF from 127.16.122.131:33649 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:46.531153 26054 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:46.526784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:50475)
0923 23:10:46.526926 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:46.526940 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:46.527013 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:46.527191 (+ 178us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:46.527192 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:46.527362 (+ 170us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:46.527412 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.527414 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.528030 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.528031 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.528571 (+ 540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.528573 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.528866 (+ 293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.528867 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.528909 (+ 42us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.529338 (+ 429us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:46.529343 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:46.531084 (+ 1741us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":65,"threads_started":1}
W20260923 23:10:46.531215 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:46.526864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50475 (local address 127.16.122.190:35905)
0923 23:10:46.526987 (+ 123us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:46.526988 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:46.526998 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:46.527107 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:46.527107 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:46.527117 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:46.527174 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:46.527470 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.527972 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.528631 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.528812 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.528856 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.531159 (+ 2303us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:50475: BlockingRecv error: recv got EOF from 127.16.122.129:50475 (error 108)
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":69,"threads_started":1}
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:46.658828 26056 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:46.654508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:38245)
0923 23:10:46.654641 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:46.654651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:46.654710 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:46.654829 (+ 119us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:46.654830 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:46.655003 (+ 173us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:46.655070 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.655072 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.655693 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.655694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.656217 (+ 523us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.656218 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.656539 (+ 321us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.656540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.656594 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.657027 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:46.657031 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:46.658746 (+ 1715us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":74,"threads_started":1}
W20260923 23:10:46.658943 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:46.654649 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:38245 (local address 127.16.122.190:35905)
0923 23:10:46.654705 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:46.654707 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:46.654715 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:46.654763 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:46.654764 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:46.654775 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:46.654819 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:46.655141 (+ 322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.655624 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.656294 (+ 670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.656481 (+ 187us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.656536 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.658878 (+ 2342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:38245: BlockingRecv error: recv got EOF from 127.16.122.130:38245 (error 108)
Metrics: {"server-negotiator.queue_time_us":35}
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:46 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:46.785465 26057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:46.781438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:60855)
0923 23:10:46.781571 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:46.781580 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:46.781636 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:46.781756 (+ 120us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:46.781756 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:46.781918 (+ 162us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:46.781966 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.781967 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.782558 (+ 591us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.782559 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.783106 (+ 547us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:46.783108 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.783384 (+ 276us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.783384 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:46.783418 (+ 34us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.783827 (+ 409us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:46.783831 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:46.785398 (+ 1567us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":77,"threads_started":1}
W20260923 23:10:46.785578 26055 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:46.781509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:60855 (local address 127.16.122.190:35905)
0923 23:10:46.781567 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:46.781568 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:46.781633 (+ 65us) server_negotiation.cc:407] Connection header received
0923 23:10:46.781693 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:46.781694 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:46.781705 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:46.781744 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:46.782017 (+ 273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.782506 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.783161 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:46.783335 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:46.783382 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:46.785508 (+ 2126us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:60855: BlockingRecv error: recv got EOF from 127.16.122.131:60855 (error 108)
Metrics: {"server-negotiator.queue_time_us":32}
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:47.536278 26058 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:47.531652 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:42385)
0923 23:10:47.531772 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:47.531783 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:47.531840 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:47.531985 (+ 145us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:47.531986 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:47.532145 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:47.532191 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.532192 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.532911 (+ 719us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.532912 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.533426 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.533427 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.533747 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.533747 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.533782 (+ 35us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.534296 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:47.534301 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:47.536218 (+ 1917us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":63,"threads_started":1}
W20260923 23:10:47.536433 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:47.531696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:42385 (local address 127.16.122.190:35905)
0923 23:10:47.531829 (+ 133us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:47.531830 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:47.531842 (+ 12us) server_negotiation.cc:407] Connection header received
0923 23:10:47.531911 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:47.531911 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:47.531921 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:47.531973 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:47.532280 (+ 307us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.532844 (+ 564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.533490 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.533691 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.533744 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.536359 (+ 2615us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:42385: BlockingRecv error: recv got EOF from 127.16.122.129:42385 (error 108)
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":72,"threads_started":1}
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:47.667016 26060 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:47.659410 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:56455)
0923 23:10:47.659520 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:47.659531 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:47.659601 (+ 70us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:47.659738 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:47.659740 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:47.659915 (+ 175us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:47.659961 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.659963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.660603 (+ 640us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.660604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.661117 (+ 513us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.661118 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.661446 (+ 328us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.661447 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.661498 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.665281 (+ 3783us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:47.665286 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:47.666936 (+ 1650us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":91,"thread_start_us":63,"threads_started":1}
W20260923 23:10:47.667117 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:47.659507 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56455 (local address 127.16.122.190:35905)
0923 23:10:47.659572 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:47.659574 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:47.659583 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:47.659664 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:47.659665 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:47.659677 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:47.659721 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:47.660038 (+ 317us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.660531 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.661194 (+ 663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.661387 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.661443 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.667064 (+ 5621us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:56455: BlockingRecv error: recv got EOF from 127.16.122.130:56455 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:47 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.131@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:47.790198 26061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:47.785892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.131:43925)
0923 23:10:47.786047 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:47.786061 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:47.786138 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:47.786270 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:47.786271 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:47.786451 (+ 180us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:47.786495 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.786497 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.787176 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.787177 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.787728 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:47.787730 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.788060 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.788061 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:47.788115 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.788509 (+ 394us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:47.788512 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:47.790111 (+ 1599us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":63,"threads_started":1}
W20260923 23:10:47.790261 26059 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:47.785997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:43925 (local address 127.16.122.190:35905)
0923 23:10:47.786054 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:47.786056 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:47.786128 (+ 72us) server_negotiation.cc:407] Connection header received
0923 23:10:47.786192 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:47.786193 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:47.786207 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:47.786259 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:47.786572 (+ 313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.787110 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.787809 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:47.788001 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:47.788056 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:47.790210 (+ 2154us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.131:43925: BlockingRecv error: recv got EOF from 127.16.122.131:43925 (error 108)
Metrics: {"server-negotiator.queue_time_us":36}
Sep 23 23:10:48 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:48 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.129@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:48.541116 26062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:48.536814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.129:44077)
0923 23:10:48.536920 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:48.536935 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:48.536992 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:48.537124 (+ 132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:48.537125 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:48.537275 (+ 150us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:48.537335 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:48.537337 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.537972 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.537973 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:48.538522 (+ 549us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:48.538524 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.538843 (+ 319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.538844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:48.538894 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:48.539333 (+ 439us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:48.539338 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:48.541046 (+ 1708us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":88,"thread_start_us":64,"threads_started":1}
W20260923 23:10:48.541204 26063 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:48.536900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44077 (local address 127.16.122.190:35905)
0923 23:10:48.537015 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:48.537017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:48.537026 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:10:48.537050 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:48.537051 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:48.537061 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:48.537116 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:48.537385 (+ 269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.537882 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.538585 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.538766 (+ 181us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.538808 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:48.541134 (+ 2326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.129:44077: BlockingRecv error: recv got EOF from 127.16.122.129:44077 (error 108)
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":58,"threads_started":1}
Sep 23 23:10:48 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
Sep 23 23:10:48 dist-test-slave-95hh krb5kdc[25003](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.122.130@KRBTEST.COM for oryx/127.16.122.190@KRBTEST.COM, Server not found in Kerberos database
I20260923 23:10:48.671527 26064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:48.667568 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:35905 (local address 127.16.122.130:57635)
0923 23:10:48.667706 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:48.667717 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:48.667774 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:48.667897 (+ 123us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:48.667898 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:48.668053 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:48.668098 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:48.668099 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.668677 (+ 578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.668678 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:48.669184 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:48.669185 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.669455 (+ 270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.669456 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:48.669492 (+ 36us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:48.669876 (+ 384us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:48.669880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:48.671471 (+ 1591us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.122.190:35905: Server oryx/127.16.122.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":62,"threads_started":1}
W20260923 23:10:48.671617 26063 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:48.667630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:57635 (local address 127.16.122.190:35905)
0923 23:10:48.667675 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:48.667677 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:48.667771 (+ 94us) server_negotiation.cc:407] Connection header received
0923 23:10:48.667830 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:48.667831 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:48.667842 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:48.667879 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:48.668149 (+ 270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.668626 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.669238 (+ 612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.669408 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.669451 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:48.671556 (+ 2105us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.122.130:57635: BlockingRecv error: recv got EOF from 127.16.122.130:57635 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:10:48.716356 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25586
I20260923 23:10:48.720247 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25738
I20260923 23:10:48.723927 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25872
I20260923 23:10:48.727267 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 25516
2026-09-23T23:10:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (17347 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-09-23T23:10:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:48Z Disabled control of system clock
I20260923 23:10:48.752391 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:33539
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:33539
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260923 23:10:48.829865 26068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:48.830091 26068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:48.830111 26068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:48.831284 26068 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:48.831321 26068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:48.831331 26068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:48.831342 26068 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:48.831355 26068 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:48.832659 26068 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:33539
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:33539
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26068
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:48.832870 26068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:48.833124 26068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:48.837980 26076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:48.837984 26074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:48.838156 26068 server_base.cc:1034] running on GCE node
W20260923 23:10:48.837984 26073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:48.838445 26068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:48.838733 26068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:48.839907 26068 hybrid_clock.cc:648] HybridClock initialized: now 1790205048839893 us; error 28 us; skew 500 ppm
I20260923 23:10:48.841127 26068 webserver.cc:460] Webserver started at http://127.16.122.190:41871/ using document root <none> and password file <none>
I20260923 23:10:48.841351 26068 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:48.841400 26068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:48.841523 26068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:48.842574 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "01b492eb38d44601a1807c9e7bcab794"
format_stamp: "Formatted at 2026-09-23 23:10:48 on dist-test-slave-95hh"
server_key: "7adcd38d9d03bfd5383173bf98475138"
server_key_iv: "2dd48fce5dca0fb0c45758abbe0d8477"
server_key_version: "encryptionkey@0"
I20260923 23:10:48.842905 26068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "01b492eb38d44601a1807c9e7bcab794"
format_stamp: "Formatted at 2026-09-23 23:10:48 on dist-test-slave-95hh"
server_key: "7adcd38d9d03bfd5383173bf98475138"
server_key_iv: "2dd48fce5dca0fb0c45758abbe0d8477"
server_key_version: "encryptionkey@0"
I20260923 23:10:48.844177 26068 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:10:48.844921 26082 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:48.845074 26068 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:48.845165 26068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "01b492eb38d44601a1807c9e7bcab794"
format_stamp: "Formatted at 2026-09-23 23:10:48 on dist-test-slave-95hh"
server_key: "7adcd38d9d03bfd5383173bf98475138"
server_key_iv: "2dd48fce5dca0fb0c45758abbe0d8477"
server_key_version: "encryptionkey@0"
I20260923 23:10:48.845235 26068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:48.871615 26068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:48.871953 26068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:48.872087 26068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:48.875726 26068 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:33539
I20260923 23:10:48.875761 26134 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:33539 every 8 connection(s)
I20260923 23:10:48.876118 26068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:48.876732 26135 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:48.879284 26135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794: Bootstrap starting.
I20260923 23:10:48.879839 26135 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:48.880115 26135 log.cc:824] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:48.880832 26135 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794: No bootstrap required, opened a new log
I20260923 23:10:48.882368 26135 raft_consensus.cc:348] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } }
I20260923 23:10:48.882509 26135 raft_consensus.cc:374] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:48.882532 26135 raft_consensus.cc:729] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 01b492eb38d44601a1807c9e7bcab794, State: Initialized, Role: FOLLOWER
I20260923 23:10:48.882674 26135 consensus_queue.cc:260] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } }
I20260923 23:10:48.882742 26135 raft_consensus.cc:388] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:48.882763 26135 raft_consensus.cc:482] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:48.882790 26135 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:48.883342 26135 raft_consensus.cc:504] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } }
I20260923 23:10:48.883426 26135 leader_election.cc:302] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 01b492eb38d44601a1807c9e7bcab794; no voters:
I20260923 23:10:48.883569 26135 leader_election.cc:288] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:48.883688 26138 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:48.883817 26135 sys_catalog.cc:564] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:48.883893 26138 raft_consensus.cc:686] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [term 1 LEADER]: Becoming Leader. State: Replica: 01b492eb38d44601a1807c9e7bcab794, State: Running, Role: LEADER
I20260923 23:10:48.883996 26138 consensus_queue.cc:237] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } }
I20260923 23:10:48.884359 26139 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "01b492eb38d44601a1807c9e7bcab794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } } }
I20260923 23:10:48.884433 26139 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:48.884502 26140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 01b492eb38d44601a1807c9e7bcab794. Latest consensus state: current_term: 1 leader_uuid: "01b492eb38d44601a1807c9e7bcab794" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "01b492eb38d44601a1807c9e7bcab794" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33539 } } }
I20260923 23:10:48.884573 26140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:48.884730 26147 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:48.885217 26147 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:48.886108 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26068
I20260923 23:10:48.886194 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:48.886380 16874 external_mini_cluster.cc:1428] Setting key 50f6f9a7b72995ff121b5995b26d7b12
I20260923 23:10:48.886757 26147 catalog_manager.cc:1347] Generated new cluster ID: e9b4999c51a54be78fede0eba80f946a
I20260923 23:10:48.886818 26147 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:48.895274 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:48.887152 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59012 (local address 127.16.122.190:33539)
0923 23:10:48.891211 (+ 4059us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:48.891220 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:48.891254 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:10:48.891353 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:48.891359 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:48.891415 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:48.891542 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:48.891760 (+ 218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.892669 (+ 909us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.893398 (+ 729us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:48.893834 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:48.893954 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:48.894591 (+ 637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:48.894595 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:48.894595 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:48.894769 (+ 174us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:48.894804 (+ 35us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:48.894815 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:48.894869 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:48.894953 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:10:48.895072 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3972,"thread_start_us":66,"threads_started":1}
I20260923 23:10:48.908393 26147 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:48.908929 26147 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:48.911653 26147 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 01b492eb38d44601a1807c9e7bcab794: Generated new TSK 0
I20260923 23:10:48.911832 26147 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:48.913669 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--builtin_ntp_servers=127.16.122.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 23:10:49.011585 26159 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.011793 26159 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.011814 26159 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.012970 26159 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.013008 26159 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.013033 26159 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:49.014318 26159 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26159
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.014552 26159 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.014832 26159 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.019618 26165 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.019621 26164 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.019639 26167 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.020123 26159 server_base.cc:1034] running on GCE node
I20260923 23:10:49.020357 26159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.020601 26159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.021776 26159 hybrid_clock.cc:648] HybridClock initialized: now 1790205049021764 us; error 29 us; skew 500 ppm
I20260923 23:10:49.023088 26159 webserver.cc:460] Webserver started at http://127.16.122.129:43669/ using document root <none> and password file <none>
I20260923 23:10:49.023315 26159 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.023370 26159 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.023492 26159 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.024518 26159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "d0ee100d74d14f8ea38b36da03dcffaa"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "4b16bb0071d25279ea097128da673cb0"
server_key_iv: "9f975ed85ba11a0b15f7e770c9e81aa8"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.024843 26159 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "d0ee100d74d14f8ea38b36da03dcffaa"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "4b16bb0071d25279ea097128da673cb0"
server_key_iv: "9f975ed85ba11a0b15f7e770c9e81aa8"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.026057 26159 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:49.026736 26173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.026890 26159 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.026991 26159 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "d0ee100d74d14f8ea38b36da03dcffaa"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "4b16bb0071d25279ea097128da673cb0"
server_key_iv: "9f975ed85ba11a0b15f7e770c9e81aa8"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.027084 26159 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.067615 26159 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.067924 26159 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.068024 26159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.068223 26159 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.068508 26159 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.068537 26159 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.068558 26159 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.068574 26159 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.073793 26159 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:41743
I20260923 23:10:49.073904 26286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:41743 every 8 connection(s)
I20260923 23:10:49.074175 26159 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:49.079053 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.074876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:48397 (local address 127.16.122.190:33539)
0923 23:10:49.074943 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.074945 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.075207 (+ 262us) server_negotiation.cc:407] Connection header received
0923 23:10:49.075699 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.075700 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.075712 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.075772 (+ 60us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.076391 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.076902 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.077829 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.078012 (+ 183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.078058 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.078827 (+ 769us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.078828 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.078828 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.078835 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.078847 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.078850 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.078890 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.078944 (+ 54us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.078965 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:10:49.079226 26289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.074870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:33539 (local address 127.16.122.129:48397)
0923 23:10:49.075078 (+ 208us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.075121 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.075615 (+ 494us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.075828 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.075831 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.075840 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.076315 (+ 475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.076319 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.076970 (+ 651us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.076971 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.077749 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.077750 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.078070 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.078071 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.078170 (+ 99us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.078685 (+ 515us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.078686 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.078715 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.078738 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.078890 (+ 152us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.078893 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.078895 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.078938 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.079020 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260923 23:10:49.079638 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26159
I20260923 23:10:49.079698 26287 heartbeater.cc:347] Connected to a master server at 127.16.122.190:33539
I20260923 23:10:49.079726 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:49.079793 26287 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.079919 16874 external_mini_cluster.cc:1428] Setting key 613c912a5bf87853c0235b02f04d169a
I20260923 23:10:49.079972 26287 heartbeater.cc:511] Master 127.16.122.190:33539 requested a full tablet report, sending...
I20260923 23:10:49.080546 26098 ts_manager.cc:194] Registered new tserver with Master: d0ee100d74d14f8ea38b36da03dcffaa (127.16.122.129:41743)
I20260923 23:10:49.080775 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--builtin_ntp_servers=127.16.122.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260923 23:10:49.081360 26098 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.129:48397
W20260923 23:10:49.159715 26290 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.159933 26290 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.159963 26290 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.161100 26290 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.161140 26290 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.161171 26290 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:49.162477 26290 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26290
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.162706 26290 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.162981 26290 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.167733 26296 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.167737 26295 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.167832 26290 server_base.cc:1034] running on GCE node
W20260923 23:10:49.167737 26298 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.168185 26290 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.168413 26290 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.169569 26290 hybrid_clock.cc:648] HybridClock initialized: now 1790205049169552 us; error 27 us; skew 500 ppm
I20260923 23:10:49.170763 26290 webserver.cc:460] Webserver started at http://127.16.122.130:43683/ using document root <none> and password file <none>
I20260923 23:10:49.170969 26290 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.171016 26290 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.171164 26290 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.172222 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "d905085805b34413a22e1ba299be91a8"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "5449ba1c77c629c6b190aa843d7be87b"
server_key_iv: "e64685fe6049a1d8dc6bc741a30592d1"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.172559 26290 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "d905085805b34413a22e1ba299be91a8"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "5449ba1c77c629c6b190aa843d7be87b"
server_key_iv: "e64685fe6049a1d8dc6bc741a30592d1"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.173687 26290 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:10:49.174299 26304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.174448 26290 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.174533 26290 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "d905085805b34413a22e1ba299be91a8"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "5449ba1c77c629c6b190aa843d7be87b"
server_key_iv: "e64685fe6049a1d8dc6bc741a30592d1"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.174594 26290 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.213342 26290 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.213660 26290 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.213793 26290 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.214011 26290 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.214341 26290 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.214373 26290 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.214402 26290 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.214419 26290 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.219702 26290 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:40769
I20260923 23:10:49.219751 26417 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:40769 every 8 connection(s)
I20260923 23:10:49.220050 26290 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:49.225099 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.220704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:53335 (local address 127.16.122.190:33539)
0923 23:10:49.220761 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.220763 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.220985 (+ 222us) server_negotiation.cc:407] Connection header received
0923 23:10:49.221460 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.221461 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.221469 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.221528 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.222212 (+ 684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.222749 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.223702 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.223897 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.223951 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.224863 (+ 912us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.224865 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.224865 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.224870 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.224884 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.224888 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.224941 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.225012 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.225037 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:49.225317 26420 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.220684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:33539 (local address 127.16.122.130:53335)
0923 23:10:49.220866 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.220906 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.221388 (+ 482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.221596 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.221600 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.221608 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.222097 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.222102 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.222825 (+ 723us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.222825 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.223635 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.223638 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.223989 (+ 351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.223990 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.224111 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.224699 (+ 588us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.224700 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.224733 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.224758 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.224950 (+ 192us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.224952 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.224955 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.225010 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.225094 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":78,"threads_started":1}
I20260923 23:10:49.225586 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26290
I20260923 23:10:49.225679 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:49.225809 26418 heartbeater.cc:347] Connected to a master server at 127.16.122.190:33539
I20260923 23:10:49.225878 16874 external_mini_cluster.cc:1428] Setting key 7e6390365dec03ec9bba80ae1751c251
I20260923 23:10:49.225908 26418 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.226089 26418 heartbeater.cc:511] Master 127.16.122.190:33539 requested a full tablet report, sending...
I20260923 23:10:49.226432 26099 ts_manager.cc:194] Registered new tserver with Master: d905085805b34413a22e1ba299be91a8 (127.16.122.130:40769)
I20260923 23:10:49.226799 26099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.130:53335
I20260923 23:10:49.226881 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--builtin_ntp_servers=127.16.122.148:41897
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260923 23:10:49.305379 26421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.305586 26421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.305606 26421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.306752 26421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.306794 26421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.306826 26421 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:49.308175 26421 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41897
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:33539
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26421
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.308401 26421 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.308686 26421 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.313463 26429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.313488 26427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.313467 26426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.313707 26421 server_base.cc:1034] running on GCE node
I20260923 23:10:49.313984 26421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.314235 26421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.315375 26421 hybrid_clock.cc:648] HybridClock initialized: now 1790205049315356 us; error 31 us; skew 500 ppm
I20260923 23:10:49.316576 26421 webserver.cc:460] Webserver started at http://127.16.122.131:42885/ using document root <none> and password file <none>
I20260923 23:10:49.316802 26421 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.316850 26421 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.316977 26421 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.318058 26421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "7157af64098e4a12ab78a70d16c55ff5"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "97b115d246944900c91900890db66a0c"
server_key_iv: "fa533552a713c8b3c7f1d6fad08ea881"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.318396 26421 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "7157af64098e4a12ab78a70d16c55ff5"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "97b115d246944900c91900890db66a0c"
server_key_iv: "fa533552a713c8b3c7f1d6fad08ea881"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.319658 26421 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:49.320272 26435 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.320406 26421 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.320484 26421 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "7157af64098e4a12ab78a70d16c55ff5"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "97b115d246944900c91900890db66a0c"
server_key_iv: "fa533552a713c8b3c7f1d6fad08ea881"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.320551 26421 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.337584 26421 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.337913 26421 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.338056 26421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.338275 26421 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.338593 26421 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.338625 26421 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.338653 26421 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.338676 26421 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.344033 26421 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:35865
I20260923 23:10:49.344105 26548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:35865 every 8 connection(s)
I20260923 23:10:49.344396 26421 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:49.349538 26158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.345076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:56705 (local address 127.16.122.190:33539)
0923 23:10:49.345135 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.345137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.345388 (+ 251us) server_negotiation.cc:407] Connection header received
0923 23:10:49.345891 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.345892 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.345903 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.345980 (+ 77us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.346647 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.347189 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.348201 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.348405 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.348462 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.349264 (+ 802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.349266 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.349266 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.349271 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.349286 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.349291 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.349335 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.349425 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.349455 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:10:49.349731 26551 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.345077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:33539 (local address 127.16.122.131:56705)
0923 23:10:49.345263 (+ 186us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.345298 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.345794 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.346054 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.346057 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.346066 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.346561 (+ 495us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.346566 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.347279 (+ 713us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.347280 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.348119 (+ 839us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:49.348122 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:49.348476 (+ 354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:49.348477 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:49.348602 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:49.349108 (+ 506us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.349108 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.349142 (+ 34us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.349167 (+ 25us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.349358 (+ 191us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.349363 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.349365 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.349424 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.349532 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":79,"threads_started":1}
I20260923 23:10:49.350219 26549 heartbeater.cc:347] Connected to a master server at 127.16.122.190:33539
I20260923 23:10:49.350306 26549 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.350478 26549 heartbeater.cc:511] Master 127.16.122.190:33539 requested a full tablet report, sending...
I20260923 23:10:49.350800 26099 ts_manager.cc:194] Registered new tserver with Master: 7157af64098e4a12ab78a70d16c55ff5 (127.16.122.131:35865)
I20260923 23:10:49.351181 26099 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.131:56705
I20260923 23:10:49.351321 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26421
I20260923 23:10:49.351432 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:49.351615 16874 external_mini_cluster.cc:1428] Setting key bd9b3ff86cbe632ae3332aa3279c4026
I20260923 23:10:49.352286 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:10:49.353873 26158 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:49.353459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59020 (local address 127.16.122.190:33539)
0923 23:10:49.353518 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.353520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.353630 (+ 110us) server_negotiation.cc:407] Connection header received
0923 23:10:49.353667 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.353667 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.353677 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.353719 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.353814 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59020: BlockingRecv error: recv got EOF from 127.0.0.1:59020 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:10:49.353943 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26159
I20260923 23:10:49.358202 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26290
I20260923 23:10:49.361846 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26421
I20260923 23:10:49.365528 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26068
2026-09-23T23:10:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (637 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-09-23T23:10:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:49Z Disabled control of system clock
I20260923 23:10:49.389868 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37099
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:37099
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:10:49.469601 26560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.469808 26560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.469828 26560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.470922 26560 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:49.470956 26560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.470968 26560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.470978 26560 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:49.470988 26560 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:49.472334 26560 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:37099
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37099
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26560
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.472535 26560 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.472774 26560 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:10:49.477535 26560 server_base.cc:1034] running on GCE node
W20260923 23:10:49.477504 26566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.477521 26568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.477469 26565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.477922 26560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.478179 26560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.479300 26560 hybrid_clock.cc:648] HybridClock initialized: now 1790205049479288 us; error 29 us; skew 500 ppm
I20260923 23:10:49.480508 26560 webserver.cc:460] Webserver started at http://127.16.122.190:37793/ using document root <none> and password file <none>
I20260923 23:10:49.480733 26560 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.480777 26560 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.480903 26560 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.481920 26560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "9cc22f8d7d364d6bba3bdca70a2a5682"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8611e9b729c3fc47b2995c5b4e84eae2"
server_key_iv: "aaf10804e16967e91ed7d96825c741ea"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.482239 26560 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "9cc22f8d7d364d6bba3bdca70a2a5682"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8611e9b729c3fc47b2995c5b4e84eae2"
server_key_iv: "aaf10804e16967e91ed7d96825c741ea"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.483467 26560 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:49.484145 26574 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.484277 26560 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.484344 26560 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "9cc22f8d7d364d6bba3bdca70a2a5682"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8611e9b729c3fc47b2995c5b4e84eae2"
server_key_iv: "aaf10804e16967e91ed7d96825c741ea"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.484411 26560 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.487681 26560 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.487931 26560 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.488045 26560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.491575 26560 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:37099
I20260923 23:10:49.491604 26626 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:37099 every 8 connection(s)
I20260923 23:10:49.491919 26560 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:49.492558 26627 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:49.493695 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26560
I20260923 23:10:49.493768 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:49.493983 16874 external_mini_cluster.cc:1428] Setting key ac3bc39d03e9d66d98b3767164aec0c8
I20260923 23:10:49.496114 26627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682: Bootstrap starting.
I20260923 23:10:49.496543 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.494529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42854 (local address 127.16.122.190:37099)
0923 23:10:49.494724 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.494734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.494765 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:10:49.495276 (+ 511us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.495285 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.495347 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.495500 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.496074 (+ 574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.496077 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.496078 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.496082 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.496127 (+ 45us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.496140 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.496210 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.496276 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.496346 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260923 23:10:49.496845 26627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:49.497097 26627 log.cc:824] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:49.497836 26627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682: No bootstrap required, opened a new log
I20260923 23:10:49.499352 26627 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } }
I20260923 23:10:49.499516 26627 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:49.499552 26627 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9cc22f8d7d364d6bba3bdca70a2a5682, State: Initialized, Role: FOLLOWER
I20260923 23:10:49.499686 26627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } }
I20260923 23:10:49.499756 26627 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:49.499791 26627 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:49.499845 26627 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:49.500512 26627 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } }
I20260923 23:10:49.500622 26627 leader_election.cc:302] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9cc22f8d7d364d6bba3bdca70a2a5682; no voters:
I20260923 23:10:49.500799 26627 leader_election.cc:288] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:49.501082 26627 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:49.501086 26632 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:49.501247 26632 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [term 1 LEADER]: Becoming Leader. State: Replica: 9cc22f8d7d364d6bba3bdca70a2a5682, State: Running, Role: LEADER
I20260923 23:10:49.501356 26632 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } }
I20260923 23:10:49.501791 26633 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } } }
I20260923 23:10:49.501878 26633 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:49.502048 26634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9cc22f8d7d364d6bba3bdca70a2a5682. Latest consensus state: current_term: 1 leader_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9cc22f8d7d364d6bba3bdca70a2a5682" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37099 } } }
I20260923 23:10:49.502101 26634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:49.502094 26641 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:49.502657 26641 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:10:49.504258 26641 catalog_manager.cc:1347] Generated new cluster ID: 2d6bebe4ef3b4b488525009626248103
I20260923 23:10:49.504333 26641 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:49.517607 26641 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:49.517802 26641 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:49.524530 26641 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9cc22f8d7d364d6bba3bdca70a2a5682: Generated new TSK 0
I20260923 23:10:49.524765 26641 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:10:49.531589 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--builtin_ntp_servers=127.16.122.148:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:10:49.619781 26651 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.619992 26651 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.620020 26651 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.621194 26651 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.621237 26651 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.621270 26651 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:49.622582 26651 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26651
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.622817 26651 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.623123 26651 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.628840 26657 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.628850 26659 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.628847 26656 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.629376 26651 server_base.cc:1034] running on GCE node
I20260923 23:10:49.629558 26651 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.629770 26651 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.630928 26651 hybrid_clock.cc:648] HybridClock initialized: now 1790205049630912 us; error 31 us; skew 500 ppm
I20260923 23:10:49.632179 26651 webserver.cc:460] Webserver started at http://127.16.122.129:41903/ using document root <none> and password file <none>
I20260923 23:10:49.632418 26651 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.632467 26651 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.632593 26651 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.633701 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "86601b4b8fad45eba30bc6ee9aa02303"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8ec50781ff0883fca5afb81494222f6a"
server_key_iv: "0bbe1a92f7d13a7a885b639745b7f7a6"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.634027 26651 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "86601b4b8fad45eba30bc6ee9aa02303"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8ec50781ff0883fca5afb81494222f6a"
server_key_iv: "0bbe1a92f7d13a7a885b639745b7f7a6"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.635270 26651 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:49.636035 26665 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.636199 26651 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.636281 26651 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "86601b4b8fad45eba30bc6ee9aa02303"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "8ec50781ff0883fca5afb81494222f6a"
server_key_iv: "0bbe1a92f7d13a7a885b639745b7f7a6"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.636349 26651 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.639633 26651 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.639883 26651 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.640000 26651 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.640183 26651 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.640463 26651 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.640496 26651 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.640520 26651 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.640543 26651 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.645854 26651 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:39967
I20260923 23:10:49.645895 26778 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:39967 every 8 connection(s)
I20260923 23:10:49.646189 26651 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:10:49.647907 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26651
I20260923 23:10:49.647997 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:49.648165 16874 external_mini_cluster.cc:1428] Setting key a4ef2dabd522a9d68f85923ebe080540
I20260923 23:10:49.649010 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--builtin_ntp_servers=127.16.122.148:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260923 23:10:49.652177 26781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.646860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37099 (local address 127.16.122.129:44197)
0923 23:10:49.647020 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.647081 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.647558 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.647748 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.647751 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.647759 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.648755 (+ 996us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.648756 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.648794 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.648866 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.651784 (+ 2918us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.651791 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.651795 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.651886 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.651959 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":99,"thread_start_us":75,"threads_started":1}
I20260923 23:10:49.652889 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.646866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:44197 (local address 127.16.122.190:37099)
0923 23:10:49.646909 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.646911 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.647169 (+ 258us) server_negotiation.cc:407] Connection header received
0923 23:10:49.647639 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.647640 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.647644 (+ 4us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.647694 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.651053 (+ 3359us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.651055 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.651055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.651060 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.651076 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.651081 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.651232 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.652797 (+ 1565us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.652813 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:10:49.655194 26779 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37099
I20260923 23:10:49.655297 26779 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.655453 26779 heartbeater.cc:511] Master 127.16.122.190:37099 requested a full tablet report, sending...
I20260923 23:10:49.656046 26591 ts_manager.cc:194] Registered new tserver with Master: 86601b4b8fad45eba30bc6ee9aa02303 (127.16.122.129:39967)
W20260923 23:10:49.733973 26782 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.734174 26782 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.734193 26782 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.735360 26782 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.735402 26782 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.735426 26782 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:49.736801 26782 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26782
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.737016 26782 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.737262 26782 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.742216 26790 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.742298 26782 server_base.cc:1034] running on GCE node
W20260923 23:10:49.742264 26788 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.742264 26787 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.742597 26782 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.742830 26782 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.743970 26782 hybrid_clock.cc:648] HybridClock initialized: now 1790205049743954 us; error 30 us; skew 500 ppm
I20260923 23:10:49.745091 26782 webserver.cc:460] Webserver started at http://127.16.122.130:41191/ using document root <none> and password file <none>
I20260923 23:10:49.745298 26782 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.745344 26782 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.745468 26782 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.746515 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "686deb5afdf74d81a6c592c2cbd120ed"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "83f247bed4a5ea9f9c75fc3e8f751f57"
server_key_iv: "1829c052c4ffffaccac59ad7e2dae12b"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.746836 26782 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "686deb5afdf74d81a6c592c2cbd120ed"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "83f247bed4a5ea9f9c75fc3e8f751f57"
server_key_iv: "1829c052c4ffffaccac59ad7e2dae12b"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.748062 26782 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:49.748682 26796 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.748826 26782 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.748909 26782 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "686deb5afdf74d81a6c592c2cbd120ed"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "83f247bed4a5ea9f9c75fc3e8f751f57"
server_key_iv: "1829c052c4ffffaccac59ad7e2dae12b"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.748965 26782 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.752202 26782 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.752419 26782 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.752504 26782 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.752645 26782 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.752905 26782 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.752936 26782 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.752961 26782 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.752988 26782 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.758222 26782 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:37621
I20260923 23:10:49.758279 26909 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:37621 every 8 connection(s)
I20260923 23:10:49.758590 26782 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:10:49.761332 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.759279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:57511 (local address 127.16.122.190:37099)
0923 23:10:49.759327 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.759329 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.759602 (+ 273us) server_negotiation.cc:407] Connection header received
0923 23:10:49.760054 (+ 452us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.760055 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.760060 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.760105 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.761161 (+ 1056us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.761162 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.761162 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.761165 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.761177 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.761181 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.761221 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.761266 (+ 45us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.761277 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:10:49.761518 26912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.759276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37099 (local address 127.16.122.130:57511)
0923 23:10:49.759486 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.759520 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.759969 (+ 449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.760165 (+ 196us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.760169 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.760176 (+ 7us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.761033 (+ 857us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.761033 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.761062 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.761091 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.761221 (+ 130us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.761224 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.761226 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.761273 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.761320 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":88,"threads_started":1}
I20260923 23:10:49.761947 26910 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37099
I20260923 23:10:49.762038 26910 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.762151 26910 heartbeater.cc:511] Master 127.16.122.190:37099 requested a full tablet report, sending...
I20260923 23:10:49.762437 26591 ts_manager.cc:194] Registered new tserver with Master: 686deb5afdf74d81a6c592c2cbd120ed (127.16.122.130:37621)
I20260923 23:10:49.767302 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26782
I20260923 23:10:49.767386 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:49.767581 16874 external_mini_cluster.cc:1428] Setting key a9d86d94fe8fc0b5b65fd614a55f357d
I20260923 23:10:49.768440 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--builtin_ntp_servers=127.16.122.148:45423
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260923 23:10:49.847129 26913 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:49.847321 26913 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:49.847339 26913 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:49.848462 26913 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:49.848495 26913 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:49.848516 26913 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:49.849759 26913 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45423
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:37099
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.26913
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:49.849943 26913 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:49.850186 26913 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:49.854910 26919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:49.854907 26918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.855006 26913 server_base.cc:1034] running on GCE node
W20260923 23:10:49.854925 26921 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:49.855356 26913 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:49.855585 26913 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:49.856735 26913 hybrid_clock.cc:648] HybridClock initialized: now 1790205049856722 us; error 35 us; skew 500 ppm
I20260923 23:10:49.857991 26913 webserver.cc:460] Webserver started at http://127.16.122.131:32791/ using document root <none> and password file <none>
I20260923 23:10:49.858220 26913 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:49.858268 26913 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:49.858398 26913 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:49.859508 26913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "b1ccf690e00140f38f518c8dae338777"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "2ad6188627e5fb81dce832de0643b118"
server_key_iv: "8cd9b66a785b2aee674040fac845e5f3"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.859829 26913 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "b1ccf690e00140f38f518c8dae338777"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "2ad6188627e5fb81dce832de0643b118"
server_key_iv: "8cd9b66a785b2aee674040fac845e5f3"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.861019 26913 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:49.861687 26927 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.861830 26913 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:49.861915 26913 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "b1ccf690e00140f38f518c8dae338777"
format_stamp: "Formatted at 2026-09-23 23:10:49 on dist-test-slave-95hh"
server_key: "2ad6188627e5fb81dce832de0643b118"
server_key_iv: "8cd9b66a785b2aee674040fac845e5f3"
server_key_version: "encryptionkey@0"
I20260923 23:10:49.861984 26913 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:49.865217 26913 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:49.865450 26913 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:49.865561 26913 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:49.865752 26913 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:49.866032 26913 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:49.866065 26913 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.866089 26913 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:49.866112 26913 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:49.871275 26913 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:36581
I20260923 23:10:49.871330 27040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:36581 every 8 connection(s)
I20260923 23:10:49.871616 26913 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:10:49.872661 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 26913
I20260923 23:10:49.872737 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:49.872895 16874 external_mini_cluster.cc:1428] Setting key 00fc32ac0dcfd1abf6c218f42c699b32
I20260923 23:10:49.874428 26630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.872227 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:33079 (local address 127.16.122.190:37099)
0923 23:10:49.872265 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.872267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.872496 (+ 229us) server_negotiation.cc:407] Connection header received
0923 23:10:49.873031 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.873032 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.873038 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.873085 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.874254 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:49.874255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:49.874256 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:10:49.874260 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:49.874275 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:10:49.874280 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:49.874322 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:49.874369 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 23:10:49.874379 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:10:49.874624 27043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:49.872207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:37099 (local address 127.16.122.131:33079)
0923 23:10:49.872375 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:49.872422 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:49.872892 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:49.873172 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:49.873177 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:49.873187 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:49.874109 (+ 922us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:10:49.874109 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:49.874155 (+ 46us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:10:49.874182 (+ 27us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:49.874324 (+ 142us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:49.874326 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:49.874328 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:10:49.874374 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:10:49.874427 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":109,"thread_start_us":64,"threads_started":1}
I20260923 23:10:49.875066 27041 heartbeater.cc:347] Connected to a master server at 127.16.122.190:37099
I20260923 23:10:49.875150 27041 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:49.875283 27041 heartbeater.cc:511] Master 127.16.122.190:37099 requested a full tablet report, sending...
I20260923 23:10:49.875581 26591 ts_manager.cc:194] Registered new tserver with Master: b1ccf690e00140f38f518c8dae338777 (127.16.122.131:36581)
I20260923 23:10:49.884461 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260923 23:10:49.886195 26630 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:10:49.885724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45464 (local address 127.16.122.190:37099)
0923 23:10:49.885781 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:49.885782 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:49.885870 (+ 88us) server_negotiation.cc:407] Connection header received
0923 23:10:49.885937 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:49.885938 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:49.885943 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:49.885988 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:49.886128 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45464: BlockingRecv error: recv got EOF from 127.0.0.1:45464 (error 108)
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:10:49.886233 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26651
I20260923 23:10:49.890164 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26782
I20260923 23:10:49.893926 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26913
I20260923 23:10:49.897639 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 26560
2026-09-23T23:10:49Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (531 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:49 dist-test-slave-95hh krb5kdc[27051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:49 dist-test-slave-95hh krb5kdc[27051](info): set up 2 sockets
Sep 23 23:10:49 dist-test-slave-95hh krb5kdc[27051](info): commencing operation
krb5kdc: starting...
W20260923 23:10:50.839785 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:50 dist-test-slave-95hh krb5kdc[27051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:10:50.950677 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45069
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:45069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:51.028817 27064 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:51.029026 27064 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:51.029049 27064 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:51.030180 27064 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:10:51.030224 27064 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:51.030236 27064 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:51.030247 27064 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:10:51.030257 27064 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:10:51.031742 27064 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:45069
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27064
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:51.031981 27064 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:51.032189 27064 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:51.036762 27072 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:51.036785 27069 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:51.036762 27070 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.037024 27064 server_base.cc:1034] running on GCE node
I20260923 23:10:51.037253 27064 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:51.037536 27064 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:51.038708 27064 hybrid_clock.cc:648] HybridClock initialized: now 1790205051038692 us; error 27 us; skew 500 ppm
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:51.041061 27064 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:10:51.041525 27064 webserver.cc:460] Webserver started at http://127.16.122.190:41409/ using document root <none> and password file <none>
I20260923 23:10:51.041723 27064 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:51.041769 27064 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:51.041893 27064 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:51.042904 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "bb1d3e974bfe4dbdb322a1888e9b8728"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "504653cf1d49d212982ebf1979c11b25"
server_key_iv: "c4f5ce7cbc63af5f6aebb5bbba736df3"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.043272 27064 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "bb1d3e974bfe4dbdb322a1888e9b8728"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "504653cf1d49d212982ebf1979c11b25"
server_key_iv: "c4f5ce7cbc63af5f6aebb5bbba736df3"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.044456 27064 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:10:51.045086 27079 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.045222 27064 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:51.045303 27064 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "bb1d3e974bfe4dbdb322a1888e9b8728"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "504653cf1d49d212982ebf1979c11b25"
server_key_iv: "c4f5ce7cbc63af5f6aebb5bbba736df3"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.045374 27064 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:51.054803 27064 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:51.056447 27064 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:51.056576 27064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:51.060182 27064 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:45069
I20260923 23:10:51.060212 27131 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:45069 every 8 connection(s)
I20260923 23:10:51.060531 27064 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:10:51.061403 27132 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:51.063647 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: Bootstrap starting.
I20260923 23:10:51.064185 27132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:51.064433 27132 log.cc:824] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:51.064862 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 27064
I20260923 23:10:51.064949 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:10:51.065146 27132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: No bootstrap required, opened a new log
I20260923 23:10:51.065152 16874 external_mini_cluster.cc:1428] Setting key 7a6c79e53763f838b204953353eb310f
I20260923 23:10:51.066915 27132 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } }
I20260923 23:10:51.067076 27132 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:51.067121 27132 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb1d3e974bfe4dbdb322a1888e9b8728, State: Initialized, Role: FOLLOWER
I20260923 23:10:51.067240 27132 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } }
I20260923 23:10:51.067298 27132 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:10:51.067322 27132 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:10:51.067364 27132 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:51.067869 27132 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } }
I20260923 23:10:51.067950 27132 leader_election.cc:302] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb1d3e974bfe4dbdb322a1888e9b8728; no voters:
I20260923 23:10:51.068089 27132 leader_election.cc:288] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:10:51.068125 27137 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:51.068267 27137 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [term 1 LEADER]: Becoming Leader. State: Replica: bb1d3e974bfe4dbdb322a1888e9b8728, State: Running, Role: LEADER
I20260923 23:10:51.068325 27132 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:10:51.068423 27137 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } }
I20260923 23:10:51.069489 27138 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb1d3e974bfe4dbdb322a1888e9b8728. Latest consensus state: current_term: 1 leader_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } } }
I20260923 23:10:51.069568 27138 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:51.069731 27139 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb1d3e974bfe4dbdb322a1888e9b8728" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45069 } } }
I20260923 23:10:51.069860 27139 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728 [sys.catalog]: This master's current role is: LEADER
I20260923 23:10:51.070029 27152 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:10:51.070447 27152 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205050, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:51.072301 27152 catalog_manager.cc:1347] Generated new cluster ID: dba5a1321374467e9a94fb352ee4369d
I20260923 23:10:51.072368 27152 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:10:51.075704 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.065627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42820 (local address 127.16.122.190:45069)
0923 23:10:51.065800 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.065809 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.065833 (+ 24us) server_negotiation.cc:407] Connection header received
0923 23:10:51.066293 (+ 460us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.066300 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.066345 (+ 45us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.066469 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:51.067106 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.068437 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.069352 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.069950 (+ 598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.070065 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.072278 (+ 2213us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:51.072282 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:51.072282 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:51.072293 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:51.074180 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.074536 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.074540 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.074540 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.074639 (+ 99us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.074807 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.074808 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.074808 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.075128 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:51.075280 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.075357 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.075499 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":80,"threads_started":1}
I20260923 23:10:51.089948 27152 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:10:51.090452 27152 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:10:51.094949 27152 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: Generated new TSK 0
I20260923 23:10:51.095141 27152 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:10:51.125541 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:51.204505 27158 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:51.204711 27158 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:51.204734 27158 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:51.205868 27158 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:51.205915 27158 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:51.205940 27158 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:10:51.207391 27158 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27158
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:51.207628 27158 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:51.207842 27158 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:51.212518 27163 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.212623 27158 server_base.cc:1034] running on GCE node
W20260923 23:10:51.212695 27164 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:51.212525 27166 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.212957 27158 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:51.213217 27158 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:51.214376 27158 hybrid_clock.cc:648] HybridClock initialized: now 1790205051214356 us; error 33 us; skew 500 ppm
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:51.216780 27158 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:10:51.217200 27158 webserver.cc:460] Webserver started at http://127.16.122.129:42043/ using document root <none> and password file <none>
I20260923 23:10:51.217393 27158 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:51.217432 27158 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:51.217525 27158 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:51.218519 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "6c0537b872fa4528a9fd44e0804449da"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "cc0179ab8d564f9ba874a5d15af4ac89"
server_key_iv: "d3c31510e94602c11359b03c95e14186"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.218817 27158 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "6c0537b872fa4528a9fd44e0804449da"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "cc0179ab8d564f9ba874a5d15af4ac89"
server_key_iv: "d3c31510e94602c11359b03c95e14186"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.220098 27158 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:51.220788 27173 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.220939 27158 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:51.221017 27158 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "6c0537b872fa4528a9fd44e0804449da"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "cc0179ab8d564f9ba874a5d15af4ac89"
server_key_iv: "d3c31510e94602c11359b03c95e14186"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.221072 27158 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:51.232142 27158 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:51.234232 27158 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:51.234342 27158 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:51.234536 27158 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:51.234781 27158 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:51.234808 27158 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.234828 27158 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:51.234839 27158 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.240206 27158 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:38781
I20260923 23:10:51.240290 27286 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:38781 every 8 connection(s)
I20260923 23:10:51.240579 27158 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:51.249536 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.241276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:51297 (local address 127.16.122.190:45069)
0923 23:10:51.241335 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.241337 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.241549 (+ 212us) server_negotiation.cc:407] Connection header received
0923 23:10:51.242086 (+ 537us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.242088 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.242101 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.242151 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:51.243120 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.243631 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.244604 (+ 973us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.244812 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.244865 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.246867 (+ 2002us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:51.246869 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:51.246869 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:51.246877 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:51.248231 (+ 1354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.248625 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.248626 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.248627 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.248677 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.248906 (+ 229us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.248907 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.248908 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.249031 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:51.249088 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.249360 (+ 272us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.249443 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:10:51.249709 27289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.241265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45069 (local address 127.16.122.129:51297)
0923 23:10:51.241429 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.241466 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.241985 (+ 519us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.242229 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.242234 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.242803 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:51.243016 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.243022 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.243713 (+ 691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.243714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.244526 (+ 812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.244529 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.244878 (+ 349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.244880 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.245006 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.245463 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:51.245473 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:51.246782 (+ 1309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:51.248307 (+ 1525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.248314 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.248316 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.248560 (+ 244us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.248733 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.248734 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.248735 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.248851 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.249086 (+ 235us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:51.249093 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:51.249230 (+ 137us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.249320 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.249484 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":81,"threads_started":1}
I20260923 23:10:51.250203 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 27158
I20260923 23:10:51.250272 27287 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45069
I20260923 23:10:51.250304 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:10:51.250373 27287 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:51.250492 16874 external_mini_cluster.cc:1428] Setting key e62b5381a77c65b1825e8ffb70de86a3
I20260923 23:10:51.250555 27287 heartbeater.cc:511] Master 127.16.122.190:45069 requested a full tablet report, sending...
I20260923 23:10:51.251132 27095 ts_manager.cc:194] Registered new tserver with Master: 6c0537b872fa4528a9fd44e0804449da (127.16.122.129:38781)
I20260923 23:10:51.251739 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:51297
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:10:51.272174 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:51.351738 27292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:51.351960 27292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:51.351984 27292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:51.353089 27292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:51.353128 27292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:51.353155 27292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:10:51.354530 27292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:51.354717 27292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:51.354908 27292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:51.359707 27298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.359856 27292 server_base.cc:1034] running on GCE node
W20260923 23:10:51.359709 27300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:51.360009 27297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.360196 27292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:51.360442 27292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:51.361603 27292 hybrid_clock.cc:648] HybridClock initialized: now 1790205051361589 us; error 32 us; skew 500 ppm
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:51.364259 27292 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:10:51.364709 27292 webserver.cc:460] Webserver started at http://127.16.122.130:42191/ using document root <none> and password file <none>
I20260923 23:10:51.364913 27292 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:51.364960 27292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:51.365087 27292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:51.366122 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "74ead2006e85453e85f560b0e84acb8d"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "295d174c4ae6a7f08ac1da1ce03f771b"
server_key_iv: "ffc4f79dcbe998fbd69c3e4281e7ea5a"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.366452 27292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "74ead2006e85453e85f560b0e84acb8d"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "295d174c4ae6a7f08ac1da1ce03f771b"
server_key_iv: "ffc4f79dcbe998fbd69c3e4281e7ea5a"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.367710 27292 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:10:51.368345 27307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.368542 27292 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:10:51.368624 27292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "74ead2006e85453e85f560b0e84acb8d"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "295d174c4ae6a7f08ac1da1ce03f771b"
server_key_iv: "ffc4f79dcbe998fbd69c3e4281e7ea5a"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.368683 27292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:51.380369 27292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:51.382465 27292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:51.382603 27292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:51.382817 27292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:51.383136 27292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:51.383172 27292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.383203 27292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:51.383225 27292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.388772 27292 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:33167
I20260923 23:10:51.388808 27420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:33167 every 8 connection(s)
I20260923 23:10:51.389110 27292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:51.396579 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 27292
I20260923 23:10:51.396680 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:10:51.396845 16874 external_mini_cluster.cc:1428] Setting key 03773d6660cc8ddaa0ebf036ca155d31
I20260923 23:10:51.397876 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.389696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:50041 (local address 127.16.122.190:45069)
0923 23:10:51.389738 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.389740 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.389922 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:10:51.390427 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.390428 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.390441 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.390492 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:51.391490 (+ 998us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.391986 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.392961 (+ 975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.393163 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.393320 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.395425 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:51.395427 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:51.395427 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:51.395435 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:51.396570 (+ 1135us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.396990 (+ 420us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.396991 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.396991 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.397048 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.397266 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.397267 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.397267 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.397385 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:51.397443 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.397707 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.397787 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:51.398075 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.389679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45069 (local address 127.16.122.130:50041)
0923 23:10:51.389805 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.389843 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.390336 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.390568 (+ 232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.390572 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.391187 (+ 615us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:51.391395 (+ 208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.391400 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.392067 (+ 667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.392069 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.392877 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.392878 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.393332 (+ 454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.393334 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.393483 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.393957 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:51.393965 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:51.395327 (+ 1362us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:51.396643 (+ 1316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.396651 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.396654 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.396928 (+ 274us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.397102 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.397102 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.397103 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.397213 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.397439 (+ 226us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:51.397445 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:51.397588 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.397679 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.397853 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":89,"thread_start_us":59,"threads_started":1}
I20260923 23:10:51.398489 27421 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45069
I20260923 23:10:51.398587 27421 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:51.398757 27421 heartbeater.cc:511] Master 127.16.122.190:45069 requested a full tablet report, sending...
I20260923 23:10:51.399129 27095 ts_manager.cc:194] Registered new tserver with Master: 74ead2006e85453e85f560b0e84acb8d (127.16.122.130:33167)
I20260923 23:10:51.399482 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:50041
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:10:51.421375 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46117
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:10:51.501669 27426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:10:51.501888 27426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:10:51.501910 27426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:10:51.503050 27426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:10:51.503088 27426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:10:51.503113 27426 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:10:51.504528 27426 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46117
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45069
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.27426
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:10:51.504737 27426 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:10:51.504958 27426 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:10:51.509882 27431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:10:51.509905 27434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.509927 27426 server_base.cc:1034] running on GCE node
W20260923 23:10:51.509882 27432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:10:51.510250 27426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:10:51.510500 27426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:10:51.511660 27426 hybrid_clock.cc:648] HybridClock initialized: now 1790205051511643 us; error 28 us; skew 500 ppm
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:10:51.514016 27426 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:10:51.514424 27426 webserver.cc:460] Webserver started at http://127.16.122.131:46603/ using document root <none> and password file <none>
I20260923 23:10:51.514628 27426 fs_manager.cc:359] Metadata directory not provided
I20260923 23:10:51.514676 27426 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:10:51.514798 27426 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:10:51.515882 27426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "24be7f650119476ba0b81a6cacc14514"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "2eb492a7db17c5ecc44a4876fc4aff74"
server_key_iv: "70370cad32bbe7e40a320db22d3b7862"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.516217 27426 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "24be7f650119476ba0b81a6cacc14514"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "2eb492a7db17c5ecc44a4876fc4aff74"
server_key_iv: "70370cad32bbe7e40a320db22d3b7862"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.517426 27426 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:10:51.518092 27441 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.518234 27426 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:10:51.518321 27426 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "24be7f650119476ba0b81a6cacc14514"
format_stamp: "Formatted at 2026-09-23 23:10:51 on dist-test-slave-95hh"
server_key: "2eb492a7db17c5ecc44a4876fc4aff74"
server_key_iv: "70370cad32bbe7e40a320db22d3b7862"
server_key_version: "encryptionkey@0"
I20260923 23:10:51.518388 27426 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:10:51.541414 27426 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:10:51.543521 27426 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:10:51.543653 27426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:10:51.543869 27426 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:10:51.544168 27426 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:10:51.544201 27426 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.544229 27426 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:10:51.544253 27426 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:10:51.549657 27426 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:45303
I20260923 23:10:51.549696 27554 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:45303 every 8 connection(s)
I20260923 23:10:51.550005 27426 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205051, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:10:51.556941 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 27426
I20260923 23:10:51.557050 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:10:51.557248 16874 external_mini_cluster.cc:1428] Setting key 049eb88df13defc6ee60625cd660d55e
I20260923 23:10:51.558622 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.550660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:47253 (local address 127.16.122.190:45069)
0923 23:10:51.550700 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.550702 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.550937 (+ 235us) server_negotiation.cc:407] Connection header received
0923 23:10:51.551507 (+ 570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.551508 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.551517 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.551565 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:51.552435 (+ 870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.552915 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.553814 (+ 899us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.553999 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.554043 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.556017 (+ 1974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:51.556018 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:51.556018 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:51.556025 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:51.557300 (+ 1275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.557761 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.557762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.557762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.557823 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.558050 (+ 227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.558051 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.558051 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.558163 (+ 112us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:51.558217 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.558461 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.558537 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:10:51.558800 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.550645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45069 (local address 127.16.122.131:47253)
0923 23:10:51.550787 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.550830 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.551425 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.551623 (+ 198us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.551626 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.552171 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:10:51.552363 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.552367 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.552979 (+ 612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.552979 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.553754 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.553756 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.554054 (+ 298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.554055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.554153 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.554627 (+ 474us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:10:51.554635 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:10:51.555935 (+ 1300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:10:51.557385 (+ 1450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.557392 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.557394 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.557693 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.557887 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:10:51.557887 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:10:51.557888 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:10:51.557996 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:10:51.558215 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:10:51.558221 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:10:51.558348 (+ 127us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.558434 (+ 86us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.558583 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":60,"threads_started":1}
I20260923 23:10:51.559190 27555 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45069
I20260923 23:10:51.559279 27555 heartbeater.cc:464] Registering TS with master...
I20260923 23:10:51.559449 27555 heartbeater.cc:511] Master 127.16.122.190:45069 requested a full tablet report, sending...
I20260923 23:10:51.559757 27095 ts_manager.cc:194] Registered new tserver with Master: 24be7f650119476ba0b81a6cacc14514 (127.16.122.131:45303)
I20260923 23:10:51.560082 27095 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:47253
I20260923 23:10:51.568796 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:10:51.575825 27136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.570094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42830 (local address 127.16.122.190:45069)
0923 23:10:51.570134 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.570136 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.570243 (+ 107us) server_negotiation.cc:407] Connection header received
0923 23:10:51.570286 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.570287 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.570296 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.570360 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:10:51.570882 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.571499 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.572166 (+ 667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.572417 (+ 251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.572466 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.573618 (+ 1152us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:10:51.573619 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:10:51.573620 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:10:51.573626 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:10:51.574742 (+ 1116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.575125 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.575126 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.575127 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.575182 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:10:51.575361 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:10:51.575362 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:10:51.575362 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:10:51.575488 (+ 126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:10:51.575571 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.575701 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.575761 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:10:51.577421 27095 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:10:51.578928 27095 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:10:51.584564 27566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.581035 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:33167 (local address 127.0.0.1:43384)
0923 23:10:51.581182 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.581190 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.581227 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.581499 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.581500 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.581502 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:51.581545 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.581547 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582854 (+ 1307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.582855 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.583611 (+ 756us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.583613 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.584350 (+ 737us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.584351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.584405 (+ 54us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.584415 (+ 10us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.584464 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.584510 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137,"mutex_wait_us":3,"thread_start_us":34,"threads_started":1}
I20260923 23:10:51.584614 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.581195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43384 (local address 127.16.122.130:33167)
0923 23:10:51.581369 (+ 174us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.581372 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.581380 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:51.581413 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.581416 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.581430 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.581500 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:51.581609 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582718 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.583686 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.584283 (+ 597us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.584365 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.584418 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.584495 (+ 77us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.584534 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":129,"thread_start_us":89,"threads_started":1}
I20260923 23:10:51.585346 27355 tablet_service.cc:1467] Processing CreateTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 (DEFAULT_TABLE table=test-table [id=79203e8501f64952a95a9d09ab4c4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:51.585584 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.581131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39706 (local address 127.16.122.131:45303)
0923 23:10:51.581327 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.581332 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.581340 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:51.581376 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.581378 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.581393 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.581619 (+ 226us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:51.581684 (+ 65us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582733 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.584797 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.585250 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.585303 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.585332 (+ 29us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.585467 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.585508 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":91,"threads_started":1}
I20260923 23:10:51.585584 27564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.580858 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:38781 (local address 127.0.0.1:50486)
0923 23:10:51.580999 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.581021 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.581078 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.581391 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.581394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.581396 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:51.581475 (+ 79us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.581478 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582477 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.582478 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.583889 (+ 1411us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.583891 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.585362 (+ 1471us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.585363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.585415 (+ 52us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.585419 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.585462 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.585508 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"mutex_wait_us":2,"thread_start_us":46,"threads_started":1}
I20260923 23:10:51.585740 27355 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:51.585773 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.580967 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50486 (local address 127.16.122.129:38781)
0923 23:10:51.581236 (+ 269us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.581242 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.581252 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:51.581290 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.581293 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.581307 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.581374 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:51.581565 (+ 191us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582380 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.583963 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.584647 (+ 684us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.584714 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.584763 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.585681 (+ 918us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.585717 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":237,"thread_start_us":84,"threads_started":1}
I20260923 23:10:51.586028 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.580847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:45303 (local address 127.0.0.1:39706)
0923 23:10:51.580980 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.581018 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.581077 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.581493 (+ 416us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.581494 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.581496 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:51.581552 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.581554 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.582819 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.582820 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.583885 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.583888 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.585320 (+ 1432us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.585321 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.585379 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.585383 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.585440 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.585482 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":110,"mutex_wait_us":4,"thread_start_us":52,"threads_started":1}
I20260923 23:10:51.586413 27221 tablet_service.cc:1467] Processing CreateTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 (DEFAULT_TABLE table=test-table [id=79203e8501f64952a95a9d09ab4c4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:51.586725 27221 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:51.586676 27489 tablet_service.cc:1467] Processing CreateTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 (DEFAULT_TABLE table=test-table [id=79203e8501f64952a95a9d09ab4c4b3f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:10:51.586968 27489 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:10:51.589164 27569 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Bootstrap starting.
I20260923 23:10:51.589246 27570 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Bootstrap starting.
I20260923 23:10:51.589748 27569 tablet_bootstrap.cc:654] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:51.589854 27570 tablet_bootstrap.cc:654] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:51.590121 27571 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Bootstrap starting.
I20260923 23:10:51.590309 27569 log.cc:824] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:51.590404 27570 log.cc:824] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:51.590682 27571 tablet_bootstrap.cc:654] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Neither blocks nor log segments found. Creating new log.
I20260923 23:10:51.591027 27569 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: No bootstrap required, opened a new log
I20260923 23:10:51.591085 27570 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: No bootstrap required, opened a new log
I20260923 23:10:51.591135 27569 ts_tablet_manager.cc:1397] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:51.591141 27570 ts_tablet_manager.cc:1397] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:51.591182 27571 log.cc:824] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Log is configured to *not* fsync() on all Append() calls
I20260923 23:10:51.591871 27571 tablet_bootstrap.cc:492] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: No bootstrap required, opened a new log
I20260923 23:10:51.591934 27571 ts_tablet_manager.cc:1397] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:10:51.592689 27570 raft_consensus.cc:348] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.592830 27570 raft_consensus.cc:374] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:51.592876 27570 raft_consensus.cc:729] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c0537b872fa4528a9fd44e0804449da, State: Initialized, Role: FOLLOWER
I20260923 23:10:51.592999 27570 consensus_queue.cc:260] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.592926 27569 raft_consensus.cc:348] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.593067 27569 raft_consensus.cc:374] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:51.593107 27569 raft_consensus.cc:729] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 74ead2006e85453e85f560b0e84acb8d, State: Initialized, Role: FOLLOWER
I20260923 23:10:51.593206 27569 consensus_queue.cc:260] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.593252 27570 ts_tablet_manager.cc:1428] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:51.593389 27287 heartbeater.cc:503] Master 127.16.122.190:45069 was elected leader, sending a full tablet report...
I20260923 23:10:51.593444 27569 ts_tablet_manager.cc:1428] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:10:51.593587 27421 heartbeater.cc:503] Master 127.16.122.190:45069 was elected leader, sending a full tablet report...
I20260923 23:10:51.593724 27571 raft_consensus.cc:348] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.593868 27571 raft_consensus.cc:374] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:10:51.594079 27571 raft_consensus.cc:729] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24be7f650119476ba0b81a6cacc14514, State: Initialized, Role: FOLLOWER
I20260923 23:10:51.594199 27571 consensus_queue.cc:260] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.594472 27555 heartbeater.cc:503] Master 127.16.122.190:45069 was elected leader, sending a full tablet report...
I20260923 23:10:51.594664 27571 ts_tablet_manager.cc:1428] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
W20260923 23:10:51.639626 27422 tablet.cc:2369] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:51.741304 27288 tablet.cc:2369] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:10:51.800511 27556 tablet.cc:2369] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:10:51.845393 27577 raft_consensus.cc:482] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:10:51.845520 27577 raft_consensus.cc:504] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.845887 27577 leader_election.cc:288] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6c0537b872fa4528a9fd44e0804449da (127.16.122.129:38781), 74ead2006e85453e85f560b0e84acb8d (127.16.122.130:33167)
I20260923 23:10:51.849547 27565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.846134 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40337 (local address 127.16.122.129:38781)
0923 23:10:51.846179 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.846181 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.846496 (+ 315us) server_negotiation.cc:407] Connection header received
0923 23:10:51.846538 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.846540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.846549 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.846594 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:51.846699 (+ 105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.847406 (+ 707us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.848673 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.849222 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.849301 (+ 79us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.849310 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.849444 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.849477 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:10:51.849563 27578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.846096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:38781 (local address 127.16.122.131:40337)
0923 23:10:51.846421 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.846431 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.846486 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.846599 (+ 113us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.846600 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.846601 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:51.846647 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.846648 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.847464 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.847465 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.848585 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.848586 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.849304 (+ 718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.849305 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.849375 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.849379 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.849429 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.849471 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":307,"mutex_wait_us":158,"thread_start_us":97,"threads_started":1}
I20260923 23:10:51.849598 27568 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.846135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36193 (local address 127.16.122.130:33167)
0923 23:10:51.846195 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.846197 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.846207 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:10:51.846261 (+ 54us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.846262 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.846271 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.846316 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:10:51.846514 (+ 198us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.847297 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.848676 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.849296 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.849352 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.849358 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.849503 (+ 145us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.849543 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:10:51.849781 27557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.846097 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:33167 (local address 127.16.122.131:36193)
0923 23:10:51.846141 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.846153 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:10:51.846196 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:10:51.846358 (+ 162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:10:51.846359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:10:51.846362 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:10:51.846432 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.846434 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.847362 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.847363 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.848572 (+ 1209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:10:51.848574 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.849370 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.849371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:10:51.849436 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.849439 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:10:51.849492 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:10:51.849522 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:10:51.849944 27241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2061a33ea0cb4443b41c5b9fe0ce06c5" candidate_uuid: "24be7f650119476ba0b81a6cacc14514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0537b872fa4528a9fd44e0804449da" is_pre_election: true
I20260923 23:10:51.850198 27241 raft_consensus.cc:2454] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24be7f650119476ba0b81a6cacc14514 in term 0.
I20260923 23:10:51.850191 27375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2061a33ea0cb4443b41c5b9fe0ce06c5" candidate_uuid: "24be7f650119476ba0b81a6cacc14514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ead2006e85453e85f560b0e84acb8d" is_pre_election: true
I20260923 23:10:51.850322 27375 raft_consensus.cc:2454] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 24be7f650119476ba0b81a6cacc14514 in term 0.
I20260923 23:10:51.850428 27443 leader_election.cc:302] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24be7f650119476ba0b81a6cacc14514, 6c0537b872fa4528a9fd44e0804449da; no voters:
I20260923 23:10:51.850561 27577 raft_consensus.cc:2781] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:10:51.850620 27577 raft_consensus.cc:482] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:10:51.850634 27577 raft_consensus.cc:3037] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:51.851357 27577 raft_consensus.cc:504] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.851473 27577 leader_election.cc:288] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [CANDIDATE]: Term 1 election: Requested vote from peers 6c0537b872fa4528a9fd44e0804449da (127.16.122.129:38781), 74ead2006e85453e85f560b0e84acb8d (127.16.122.130:33167)
I20260923 23:10:51.851629 27375 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2061a33ea0cb4443b41c5b9fe0ce06c5" candidate_uuid: "24be7f650119476ba0b81a6cacc14514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "74ead2006e85453e85f560b0e84acb8d"
I20260923 23:10:51.851656 27241 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2061a33ea0cb4443b41c5b9fe0ce06c5" candidate_uuid: "24be7f650119476ba0b81a6cacc14514" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c0537b872fa4528a9fd44e0804449da"
I20260923 23:10:51.851722 27375 raft_consensus.cc:3037] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:51.851722 27241 raft_consensus.cc:3037] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:10:51.852568 27241 raft_consensus.cc:2454] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24be7f650119476ba0b81a6cacc14514 in term 1.
I20260923 23:10:51.852568 27375 raft_consensus.cc:2454] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 24be7f650119476ba0b81a6cacc14514 in term 1.
I20260923 23:10:51.852736 27444 leader_election.cc:302] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 24be7f650119476ba0b81a6cacc14514, 74ead2006e85453e85f560b0e84acb8d; no voters:
I20260923 23:10:51.852813 27577 raft_consensus.cc:2781] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:10:51.852962 27577 raft_consensus.cc:686] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 1 LEADER]: Becoming Leader. State: Replica: 24be7f650119476ba0b81a6cacc14514, State: Running, Role: LEADER
I20260923 23:10:51.853055 27577 consensus_queue.cc:237] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } }
I20260923 23:10:51.853809 27096 catalog_manager.cc:5495] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 reported cstate change: term changed from 0 to 1, leader changed from <none> to 24be7f650119476ba0b81a6cacc14514 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "24be7f650119476ba0b81a6cacc14514" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "24be7f650119476ba0b81a6cacc14514" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 45303 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 } health_report { overall_health: UNKNOWN } } }
I20260923 23:10:51.865401 27567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.863319 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39712 (local address 127.16.122.131:45303)
0923 23:10:51.863368 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:10:51.863370 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:10:51.863378 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:10:51.863441 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:10:51.863442 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:10:51.863458 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:10:51.863519 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:10:51.863610 (+ 91us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.864066 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.864815 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:10:51.864991 (+ 176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:10:51.865031 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:10:51.865134 (+ 103us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:10:51.865231 (+ 97us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:10:51.865286 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:10:51.865297 (+ 11us) server_negotiation.cc:299] Negotiation successful
0923 23:10:51.865311 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:10:51.867153 27241 raft_consensus.cc:1261] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 6c0537b872fa4528a9fd44e0804449da [term 1 FOLLOWER]: Refusing update from remote peer 24be7f650119476ba0b81a6cacc14514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:51.867170 27375 raft_consensus.cc:1261] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 1 FOLLOWER]: Refusing update from remote peer 24be7f650119476ba0b81a6cacc14514: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:10:51.867486 27579 consensus_queue.cc:1035] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "74ead2006e85453e85f560b0e84acb8d" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33167 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:51.867625 27577 consensus_queue.cc:1035] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c0537b872fa4528a9fd44e0804449da" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 38781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:10:51.869906 27582 mvcc.cc:205] Tried to move back new op lower bound from 7332679892445077504 to 7332679892390526976. Current Snapshot: MvccSnapshot[applied={T|T < 7332679892445077504}]
I20260923 23:10:51.870765 27584 mvcc.cc:205] Tried to move back new op lower bound from 7332679892445077504 to 7332679892390526976. Current Snapshot: MvccSnapshot[applied={T|T < 7332679892445077504}]
I20260923 23:10:51.872059 27095 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:51.872149 27095 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42830:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:10:51.872792 27095 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bb1d3e974bfe4dbdb322a1888e9b8728: Sending DeleteTablet for 3 replicas of tablet 2061a33ea0cb4443b41c5b9fe0ce06c5
I20260923 23:10:51.873126 27489 tablet_service.cc:1514] Processing DeleteTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:51 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:39706
I20260923 23:10:51.873123 27221 tablet_service.cc:1514] Processing DeleteTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:51 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:50486
I20260923 23:10:51.873346 27355 tablet_service.cc:1514] Processing DeleteTablet for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:10:51 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:43384
I20260923 23:10:51.873426 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 27158
I20260923 23:10:51.873533 27594 tablet_replica.cc:330] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: stopping tablet replica
I20260923 23:10:51.873636 27594 raft_consensus.cc:2229] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:10:51.873734 27594 raft_consensus.cc:2258] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:51.875603 27594 ts_tablet_manager.cc:1905] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:51.876151 27592 tablet_replica.cc:330] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: stopping tablet replica
I20260923 23:10:51.876304 27592 raft_consensus.cc:2229] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:10:51.876451 27592 raft_consensus.cc:2258] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:10:51.876881 27592 ts_tablet_manager.cc:1905] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:10:51.877426 27594 ts_tablet_manager.cc:1918] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:51.877511 27594 log.cc:1196] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/2061a33ea0cb4443b41c5b9fe0ce06c5
I20260923 23:10:51.877758 27594 ts_tablet_manager.cc:1939] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 74ead2006e85453e85f560b0e84acb8d: Deleting consensus metadata
I20260923 23:10:51.878178 27082 catalog_manager.cc:4841] TS 74ead2006e85453e85f560b0e84acb8d (127.16.122.130:33167): tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 (table test-table [id=79203e8501f64952a95a9d09ab4c4b3f]) successfully deleted
I20260923 23:10:51.878311 27592 ts_tablet_manager.cc:1918] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:10:51.878376 27592 log.cc:1196] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/2061a33ea0cb4443b41c5b9fe0ce06c5
I20260923 23:10:51.878590 27592 ts_tablet_manager.cc:1939] T 2061a33ea0cb4443b41c5b9fe0ce06c5 P 24be7f650119476ba0b81a6cacc14514: Deleting consensus metadata
I20260923 23:10:51.879024 27082 catalog_manager.cc:4841] TS 24be7f650119476ba0b81a6cacc14514 (127.16.122.131:45303): tablet 2061a33ea0cb4443b41c5b9fe0ce06c5 (table test-table [id=79203e8501f64952a95a9d09ab4c4b3f]) successfully deleted
W20260923 23:10:51.881623 27081 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:38781 (error 108)
I20260923 23:10:51.881996 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 27292
I20260923 23:10:51.882099 27563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:10:51.881984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:38781 (local address 127.0.0.1:50502)
0923 23:10:51.882030 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:10:51.882065 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:38781: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:10:51.883095 27081 catalog_manager.cc:4584] TS 6c0537b872fa4528a9fd44e0804449da (127.16.122.129:38781): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2061a33ea0cb4443b41c5b9fe0ce06c5: Network error: Client connection negotiation failed: client connection to 127.16.122.129:38781: connect: Connection refused (error 111)
I20260923 23:10:51.889153 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 27426
I20260923 23:10:51.893841 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 27064
2026-09-23T23:10:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (1997 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27598](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27598](info): set up 2 sockets
Sep 23 23:10:51 dist-test-slave-95hh krb5kdc[27598](info): commencing operation
krb5kdc: starting...
W20260923 23:10:52.843799 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:10:52 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:10:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:10:52Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangeradmin_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangeradmin_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangerlookup_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangerlookup_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
I20260923 23:10:52.997011 16874 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskODLJMS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/postgres -l logfile start
2026-09-23 23:10:55.365 UTC [27628] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 23:10:55.365 UTC [27628] LOG: listening on IPv4 address "127.16.122.148", port 39437
2026-09-23 23:10:55.368 UTC [27628] LOG: listening on Unix socket "/tmp/.s.PGSQL.39437"
2026-09-23 23:10:55.395 UTC [27631] LOG: database system was shut down at 2026-09-23 23:10:54 UTC
2026-09-23 23:10:55.399 UTC [27628] LOG: database system is ready to accept connections
I20260923 23:10:56.267693 16874 mini_postgres.cc:96] Postgres bound to 39437
2026-09-23 23:10:56.273 UTC [27639] FATAL: database "slave" does not exist
127.16.122.148:39437 - accepting connections
I20260923 23:10:56.273783 16874 mini_ranger.cc:161] Starting Ranger...
I20260923 23:10:56.287401 16874 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 23:10:56.720929 16874 mini_ranger.cc:87] Created ranger Postgres database
I20260923 23:10:56.721040 16874 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin
2026-09-23 23:10:56,981 [I] DB FLAVOR :POSTGRES
2026-09-23 23:10:56,982 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:10:56,982 [I] Checking connection..
2026-09-23 23:10:56,982 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 23:10:57,239 [I] Checking connection passed.
2026-09-23 23:10:57,240 [I] --------- Verifying version history table ---------
2026-09-23 23:10:57,240 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:10:57,497 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:10:57,497 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:10:57,730 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:10:57,730 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 23:10:57,730 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 23:10:57.966 UTC [27703] WARNING: there is no transaction in progress
2026-09-23 23:10:57,972 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 23:10:57,972 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:10:58,209 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 23:10:58,209 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:10:58,209 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 23:10:58,460 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 23:10:58,704 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-95hh.c.gcp-upstream.internal','N') ;"
2026-09-23 23:10:58,917 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:10:59,171 [I] Table x_portal_user does not exist in database ranger
2026-09-23 23:10:59,172 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:10:59,416 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 23:10:59,416 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:10:59,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:10:59,905 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 23:10:59,906 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 23:11:00.851 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.867 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.885 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.901 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.917 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.935 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.988 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:00.994 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.008 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.018 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.028 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.041 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.051 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.057 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.063 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01.069 UTC [27811] WARNING: there is no transaction in progress
2026-09-23 23:11:01,655 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 23:11:01,655 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 23:11:01,896 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 23:11:01,897 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:11:02,149 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 23:11:02,150 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:11:02,406 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 23:11:02,406 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:11:02,648 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:11:02,876 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-95hh.c.gcp-upstream.internal';"
2026-09-23 23:11:03,092 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 23:11:03,092 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 23:11:03,346 [I] DB_PATCHES have already been applied
I20260923 23:11:03.351557 16874 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskODLJMS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 23:11:03.351626 16874 mini_ranger.cc:193] Using host: 127.16.122.148
I20260923 23:11:03.354518 16874 mini_ranger.cc:227] Ranger admin URL: http://127.16.122.148:44003
Sep 23, 2026 11:11:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:11:03 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:11:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 11:11:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 11:11:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 11:11:04 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 11:11:04 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 23:11:04.357890 16874 mini_ranger.cc:160] Time spent starting Ranger: real 8.084s user 0.001s sys 0.007s
Sep 23, 2026 11:11:22 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 11:11:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 11:11:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 11:11:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 11:11:23 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 23:11:24.509438 16874 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangerkms_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/rangerkms_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.122.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.122.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 23:11:24 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 23:11:24.584896 16874 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 23:11:24.593318 16874 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 23:11:25.407737 16874 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 23:11:25.407804 16874 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms
I20260923 23:11:25.407822 16874 mini_ranger_kms.cc:227] Using postgres at 127.16.122.148:39437
2026-09-23 23:11:25,550 [I] DB FLAVOR :POSTGRES
2026-09-23 23:11:25,551 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:11:25,551 [I] Checking connection
2026-09-23 23:11:25,551 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 23:11:25,784 [I] connection success
2026-09-23 23:11:25,784 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 23:11:25,784 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 23:11:25,784 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 23:11:26,031 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 23:11:26,031 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:11:26,031 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 23:11:26,031 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:39437/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 23:11:26,305 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 23:11:26.765486 16874 mini_ranger_kms.cc:310] Created kms service
I20260923 23:11:26.936452 16874 mini_ranger_kms.cc:326] Created kudu user
I20260923 23:11:26.975626 16874 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 23:11:26 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205084, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:11:27.353667 16874 mini_ranger_kms.cc:384] Added ranger policy
I20260923 23:11:27.353792 16874 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms:/tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 23:11:27.353811 16874 mini_ranger_kms.cc:240] Using host: 127.16.122.148
I20260923 23:11:27.357812 16874 mini_ranger_kms.cc:276] Ranger KMS PID: 27991
I20260923 23:11:27.357934 16874 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.122.148:40093
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 11:11:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 23:11:29.314713 16874 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.730s user 0.003s sys 0.124s
I20260923 23:11:29.314831 16874 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 23:11:29.314918 16874 mini_ranger_kms.cc:401] 127.16.122.148:40093/kms/v1/keys
Sep 23 23:11:29 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790205089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 23:11:29 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
Sep 23 23:11:30 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205089, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
Sep 23 23:11:32 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:11:32.662477 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45281
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:45281
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:40093/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:11:32.743731 28042 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:11:32.743939 28042 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:11:32.743959 28042 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:11:32.745060 28042 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:11:32.745097 28042 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:11:32.745110 28042 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:11:32.745121 28042 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:11:32.745132 28042 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:11:32.746613 28042 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.122.148:40093/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:45281
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28042
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:11:32.746798 28042 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:11:32.746996 28042 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:11:32.751713 28042 server_base.cc:1034] running on GCE node
W20260923 23:11:32.751653 28050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:32.751693 28048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:32.751693 28047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:11:32.752108 28042 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:11:32.752379 28042 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:11:32.753558 28042 hybrid_clock.cc:648] HybridClock initialized: now 1790205092753537 us; error 36 us; skew 500 ppm
Sep 23 23:11:32 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205092, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:11:32.758970 28042 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:11:32.759418 28042 webserver.cc:460] Webserver started at http://127.16.122.190:41759/ using document root <none> and password file <none>
I20260923 23:11:32.759606 28042 fs_manager.cc:359] Metadata directory not provided
I20260923 23:11:32.759641 28042 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:11:32.759732 28042 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:11:32 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205092, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:11:32.804273 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "cab18a7dc4164a5fbb88e25f4323af14"
format_stamp: "Formatted at 2026-09-23 23:11:32 on dist-test-slave-95hh"
server_key: "35cc7e1971f492dc3a100dc5495c5cc2"
server_key_iv: "66fcbc48ac630260e21bd603382aa6f5"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:32.804651 28042 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "cab18a7dc4164a5fbb88e25f4323af14"
format_stamp: "Formatted at 2026-09-23 23:11:32 on dist-test-slave-95hh"
server_key: "35cc7e1971f492dc3a100dc5495c5cc2"
server_key_iv: "66fcbc48ac630260e21bd603382aa6f5"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:32.805989 28042 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:11:32.817672 28057 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:32.817891 28042 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:11:32.817987 28042 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "cab18a7dc4164a5fbb88e25f4323af14"
format_stamp: "Formatted at 2026-09-23 23:11:32 on dist-test-slave-95hh"
server_key: "35cc7e1971f492dc3a100dc5495c5cc2"
server_key_iv: "66fcbc48ac630260e21bd603382aa6f5"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:32.818087 28042 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:11:32.832896 28042 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:11:32.834756 28042 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:11:32.834897 28042 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:11:32.838989 28042 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:45281
I20260923 23:11:32.839066 28109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:45281 every 8 connection(s)
I20260923 23:11:32.839411 28042 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:11:32.840178 28110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:11:32.843284 28110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: Bootstrap starting.
I20260923 23:11:32.843954 28110 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: Neither blocks nor log segments found. Creating new log.
I20260923 23:11:32.844259 28110 log.cc:824] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: Log is configured to *not* fsync() on all Append() calls
I20260923 23:11:32.845031 28110 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: No bootstrap required, opened a new log
I20260923 23:11:32.847817 28110 raft_consensus.cc:348] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } }
I20260923 23:11:32.848012 28110 raft_consensus.cc:374] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:11:32.848042 28110 raft_consensus.cc:729] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cab18a7dc4164a5fbb88e25f4323af14, State: Initialized, Role: FOLLOWER
I20260923 23:11:32.848179 28110 consensus_queue.cc:260] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } }
I20260923 23:11:32.848266 28110 raft_consensus.cc:388] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:11:32.848328 28110 raft_consensus.cc:482] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:11:32.848388 28110 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:11:32.848425 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 28042
I20260923 23:11:32.848536 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:11:32.848673 16874 external_mini_cluster.cc:1428] Setting key 1fe654335bdeb8f6103a27ef637676e8
I20260923 23:11:32.849071 28110 raft_consensus.cc:504] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } }
I20260923 23:11:32.849237 28110 leader_election.cc:302] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cab18a7dc4164a5fbb88e25f4323af14; no voters:
I20260923 23:11:32.849442 28110 leader_election.cc:288] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:11:32.849484 28115 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:11:32.849804 28110 sys_catalog.cc:564] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:11:32.850354 28115 raft_consensus.cc:686] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [term 1 LEADER]: Becoming Leader. State: Replica: cab18a7dc4164a5fbb88e25f4323af14, State: Running, Role: LEADER
I20260923 23:11:32.850447 28115 consensus_queue.cc:237] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } }
I20260923 23:11:32.850735 28115 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cab18a7dc4164a5fbb88e25f4323af14. Latest consensus state: current_term: 1 leader_uuid: "cab18a7dc4164a5fbb88e25f4323af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } } }
I20260923 23:11:32.850791 28115 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [sys.catalog]: This master's current role is: LEADER
I20260923 23:11:32.851857 28110 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 23:11:32.852383 28121 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:11:32.852226 28116 sys_catalog.cc:455] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cab18a7dc4164a5fbb88e25f4323af14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cab18a7dc4164a5fbb88e25f4323af14" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45281 } } }
I20260923 23:11:32.852607 28116 sys_catalog.cc:458] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14 [sys.catalog]: This master's current role is: LEADER
I20260923 23:11:32.856547 28121 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:11:32 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205092, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:11:32.859710 28121 catalog_manager.cc:1347] Generated new cluster ID: bdc9bd4370ef4efa8185d89191426a8d
I20260923 23:11:32.859782 28121 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:11:32.862905 28114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:32.849124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49838 (local address 127.16.122.190:45281)
0923 23:11:32.849325 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:32.849334 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:32.849367 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:11:32.849477 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:32.849484 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:32.849559 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:32.849704 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:11:32.850412 (+ 708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:32.851679 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:32.852662 (+ 983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:32.855505 (+ 2843us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:32.855742 (+ 237us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:32.858493 (+ 2751us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:11:32.858498 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:11:32.858498 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:11:32.858511 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:11:32.860804 (+ 2293us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:32.861219 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:32.861222 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:32.861222 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:32.861336 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:32.861536 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:32.861537 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:32.861537 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:32.861846 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:11:32.861982 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:32.862539 (+ 557us) server_negotiation.cc:299] Negotiation successful
0923 23:11:32.862669 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":155,"thread_start_us":91,"threads_started":1}
I20260923 23:11:32.866448 28121 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:11:32.867255 28121 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:11:32.876395 28121 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: Generated new TSK 0
I20260923 23:11:32.876724 28121 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:11:33.613245 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:40093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:11:33.696213 28155 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:11:33.696458 28155 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:11:33.696491 28155 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:11:33.698163 28155 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:11:33.698261 28155 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:11:33.698319 28155 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:11:33.700052 28155 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.122.148:40093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28155
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:11:33.700278 28155 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:11:33.700508 28155 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:11:33.705248 28163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:33.705351 28160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:33.705250 28161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:11:33.706068 28155 server_base.cc:1034] running on GCE node
I20260923 23:11:33.706274 28155 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:11:33.706494 28155 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:11:33.707690 28155 hybrid_clock.cc:648] HybridClock initialized: now 1790205093707679 us; error 35 us; skew 500 ppm
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:11:33.710275 28155 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:11:33.710741 28155 webserver.cc:460] Webserver started at http://127.16.122.129:45809/ using document root <none> and password file <none>
I20260923 23:11:33.711001 28155 fs_manager.cc:359] Metadata directory not provided
I20260923 23:11:33.711091 28155 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:11:33.711236 28155 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:11:33.723919 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "43cb6c4b2db94eb2814b90928cedcbf4"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "c15f851529f6d819471b25d7f02460b2"
server_key_iv: "54e8e6b40a3b8c7645acc8e7aefc64f4"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.724495 28155 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "43cb6c4b2db94eb2814b90928cedcbf4"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "c15f851529f6d819471b25d7f02460b2"
server_key_iv: "54e8e6b40a3b8c7645acc8e7aefc64f4"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.727180 28155 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 23:11:33.743177 28170 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.743377 28155 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260923 23:11:33.743470 28155 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "43cb6c4b2db94eb2814b90928cedcbf4"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "c15f851529f6d819471b25d7f02460b2"
server_key_iv: "54e8e6b40a3b8c7645acc8e7aefc64f4"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.743605 28155 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:11:33.754870 28155 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:11:33.757822 28155 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:11:33.758030 28155 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:11:33.758333 28155 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:11:33.758697 28155 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:11:33.758744 28155 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.758774 28155 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:11:33.758826 28155 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.765853 28155 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:41961
I20260923 23:11:33.766304 28155 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:11:33.767241 28285 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:41961 every 8 connection(s)
I20260923 23:11:33.768563 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 28155
I20260923 23:11:33.768671 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:11:33.768846 16874 external_mini_cluster.cc:1428] Setting key eb75af3f03dcf2336d310ffdda0e4a98
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
I20260923 23:11:33.783423 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:33.767849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:43323 (local address 127.16.122.190:45281)
0923 23:11:33.768945 (+ 1096us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:33.768948 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:33.772185 (+ 3237us) server_negotiation.cc:407] Connection header received
0923 23:11:33.772880 (+ 695us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:33.772881 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:33.772895 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:33.772977 (+ 82us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:11:33.773751 (+ 774us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:33.774552 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:33.775956 (+ 1404us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:33.776262 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:33.776342 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:33.778786 (+ 2444us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:11:33.778787 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:11:33.778787 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:11:33.778793 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:11:33.780146 (+ 1353us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:33.780598 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:33.780599 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:33.780599 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:33.780653 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:33.780897 (+ 244us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:33.780898 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:33.780898 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:33.781026 (+ 128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:11:33.781089 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:33.783238 (+ 2149us) server_negotiation.cc:299] Negotiation successful
0923 23:11:33.783323 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1060,"thread_start_us":79,"threads_started":1}
I20260923 23:11:33.783891 28289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:33.767838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45281 (local address 127.16.122.129:43323)
0923 23:11:33.772025 (+ 4187us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:33.772079 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:33.772759 (+ 680us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:33.773116 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:33.773121 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:33.773369 (+ 248us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:11:33.773640 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:33.773646 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:33.774654 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:33.774655 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:33.775860 (+ 1205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:33.775863 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:33.776353 (+ 490us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:33.776354 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:33.776493 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:33.777083 (+ 590us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:11:33.777093 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:11:33.778678 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:11:33.780224 (+ 1546us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:33.780231 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:33.780233 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:33.780527 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:33.780707 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:33.780708 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:33.780708 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:33.780842 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:33.782879 (+ 2037us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:11:33.782886 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:11:33.783066 (+ 180us) client_negotiation.cc:769] Sending connection context
0923 23:11:33.783505 (+ 439us) client_negotiation.cc:240] Negotiation successful
0923 23:11:33.783660 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4134,"thread_start_us":81,"threads_started":1}
I20260923 23:11:33.784446 28286 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45281
I20260923 23:11:33.784546 28286 heartbeater.cc:464] Registering TS with master...
I20260923 23:11:33.784742 28286 heartbeater.cc:511] Master 127.16.122.190:45281 requested a full tablet report, sending...
I20260923 23:11:33.785328 28074 ts_manager.cc:194] Registered new tserver with Master: 43cb6c4b2db94eb2814b90928cedcbf4 (127.16.122.129:41961)
I20260923 23:11:33.786206 28074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:43323
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:11:33.797578 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:40093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:11:33.905042 28292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:11:33.905313 28292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:11:33.905346 28292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:11:33.907107 28292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:11:33.907226 28292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:11:33.907305 28292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:11:33.909487 28292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.122.148:40093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28292
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:11:33.909897 28292 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:11:33.910205 28292 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:11:33.916437 28300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:33.916597 28299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:33.916429 28302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:11:33.917155 28292 server_base.cc:1034] running on GCE node
I20260923 23:11:33.917392 28292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:11:33.917677 28292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:11:33.918816 28292 hybrid_clock.cc:648] HybridClock initialized: now 1790205093918800 us; error 33 us; skew 500 ppm
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:11:33.921247 28292 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:11:33.921725 28292 webserver.cc:460] Webserver started at http://127.16.122.130:43553/ using document root <none> and password file <none>
I20260923 23:11:33.921957 28292 fs_manager.cc:359] Metadata directory not provided
I20260923 23:11:33.922014 28292 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:11:33.922174 28292 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:11:33 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:11:33.934923 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "ba91e992d1674b229a3a314fe5b6789f"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "94a297a9e763ff03440f1c77606adecb"
server_key_iv: "b2da8bae06b8af746ea593f8e9f00b3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.935519 28292 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "ba91e992d1674b229a3a314fe5b6789f"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "94a297a9e763ff03440f1c77606adecb"
server_key_iv: "b2da8bae06b8af746ea593f8e9f00b3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.946399 28292 fs_manager.cc:691] Time spent creating directory manager: real 0.011s user 0.001s sys 0.000s
I20260923 23:11:33.964845 28309 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.967872 28292 fs_manager.cc:725] Time spent opening block manager: real 0.007s user 0.000s sys 0.001s
I20260923 23:11:33.968019 28292 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "ba91e992d1674b229a3a314fe5b6789f"
format_stamp: "Formatted at 2026-09-23 23:11:33 on dist-test-slave-95hh"
server_key: "94a297a9e763ff03440f1c77606adecb"
server_key_iv: "b2da8bae06b8af746ea593f8e9f00b3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:33.968139 28292 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:11:33.984403 28292 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:11:33.986713 28292 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:11:33.986866 28292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:11:33.987115 28292 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:11:33.987504 28292 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:11:33.987566 28292 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.987602 28292 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:11:33.987622 28292 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:33.993664 28292 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:41833
I20260923 23:11:33.994103 28292 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:11:33.996783 28422 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:41833 every 8 connection(s)
I20260923 23:11:34.003290 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 28292
I20260923 23:11:34.003401 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:11:34.003582 16874 external_mini_cluster.cc:1428] Setting key be88bd83cd49d5296e25365d4a40f4e1
Sep 23 23:11:34 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205093, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
I20260923 23:11:34.015134 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:33.997714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:57655 (local address 127.16.122.190:45281)
0923 23:11:33.997782 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:33.997784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:33.998223 (+ 439us) server_negotiation.cc:407] Connection header received
0923 23:11:33.999002 (+ 779us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:33.999004 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:33.999019 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:33.999093 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:11:34.000045 (+ 952us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.000811 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.002323 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.002624 (+ 301us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.002672 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.011622 (+ 8950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:11:34.011625 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:11:34.011625 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:11:34.011635 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:11:34.013385 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.013933 (+ 548us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.013934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.013935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.013992 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.014295 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.014296 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.014296 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.014447 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:11:34.014489 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.014901 (+ 412us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.015000 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:11:34.015318 28425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:33.997388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45281 (local address 127.16.122.130:57655)
0923 23:11:33.998077 (+ 689us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:33.998128 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:33.998869 (+ 741us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:33.999352 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:33.999357 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:33.999625 (+ 268us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:11:33.999919 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:33.999926 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.000925 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.000926 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.002220 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.002223 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.002765 (+ 542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.002766 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.002908 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.006202 (+ 3294us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:11:34.006214 (+ 12us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:11:34.011451 (+ 5237us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:11:34.013494 (+ 2043us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:34.013505 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:34.013509 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:34.013846 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:34.014056 (+ 210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:34.014057 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:34.014058 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:34.014218 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:34.014533 (+ 315us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:11:34.014540 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:11:34.014731 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.014850 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.015015 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":98,"threads_started":1}
I20260923 23:11:34.016193 28423 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45281
I20260923 23:11:34.016307 28423 heartbeater.cc:464] Registering TS with master...
I20260923 23:11:34.016525 28423 heartbeater.cc:511] Master 127.16.122.190:45281 requested a full tablet report, sending...
I20260923 23:11:34.016956 28074 ts_manager.cc:194] Registered new tserver with Master: ba91e992d1674b229a3a314fe5b6789f (127.16.122.130:41833)
I20260923 23:11:34.017477 28074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:57655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:11:34.045351 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:40093/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:34093
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:11:34.166011 28429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:11:34.166306 28429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:11:34.166344 28429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:11:34.168201 28429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:11:34.168285 28429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:11:34.168326 28429 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:11:34.170461 28429 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:34093
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.122.148:40093/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45281
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.28429
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:11:34.170815 28429 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:11:34.171119 28429 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:11:34.180115 28437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:34.180315 28434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:11:34.180498 28435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:11:34.181594 28429 server_base.cc:1034] running on GCE node
I20260923 23:11:34.181870 28429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:11:34.182157 28429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:11:34.183326 28429 hybrid_clock.cc:648] HybridClock initialized: now 1790205094183304 us; error 36 us; skew 500 ppm
Sep 23 23:11:34 dist-test-slave-95hh krb5kdc[27598](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205094, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:11:34.186240 28429 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:11:34.186683 28429 webserver.cc:460] Webserver started at http://127.16.122.131:38319/ using document root <none> and password file <none>
I20260923 23:11:34.186899 28429 fs_manager.cc:359] Metadata directory not provided
I20260923 23:11:34.186947 28429 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:11:34.187099 28429 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:11:34 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205094, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:11:34.211376 28429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "248b0fbfb6504f53830930c33b923dc0"
format_stamp: "Formatted at 2026-09-23 23:11:34 on dist-test-slave-95hh"
server_key: "44a24dd89122db907c07b3909ab028b3"
server_key_iv: "35d4992fbd2a87f03c3f4fb64fcd0a3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:34.211839 28429 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "248b0fbfb6504f53830930c33b923dc0"
format_stamp: "Formatted at 2026-09-23 23:11:34 on dist-test-slave-95hh"
server_key: "44a24dd89122db907c07b3909ab028b3"
server_key_iv: "35d4992fbd2a87f03c3f4fb64fcd0a3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:34.213243 28429 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:11:34.226009 28444 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:34.226198 28429 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:11:34.226287 28429 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "248b0fbfb6504f53830930c33b923dc0"
format_stamp: "Formatted at 2026-09-23 23:11:34 on dist-test-slave-95hh"
server_key: "44a24dd89122db907c07b3909ab028b3"
server_key_iv: "35d4992fbd2a87f03c3f4fb64fcd0a3a"
server_key_version: "kuduclusterkey@0"
I20260923 23:11:34.226387 28429 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:11:34.251865 28429 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:11:34.254115 28429 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:11:34.254256 28429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:11:34.254487 28429 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:11:34.254869 28429 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:11:34.254925 28429 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:34.254954 28429 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:11:34.254977 28429 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:11:34.261101 28429 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:36567
I20260923 23:11:34.261528 28429 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:11:34.264233 28557 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:36567 every 8 connection(s)
I20260923 23:11:34.269693 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 28429
I20260923 23:11:34.269799 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:11:34.270365 16874 external_mini_cluster.cc:1428] Setting key 6e8867f2bb08f1ba562d99bab09a0299
Sep 23 23:11:34 dist-test-slave-95hh krb5kdc[27598](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205094, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:11:34.275764 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.264921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:52135 (local address 127.16.122.190:45281)
0923 23:11:34.264966 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.264968 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.265210 (+ 242us) server_negotiation.cc:407] Connection header received
0923 23:11:34.265890 (+ 680us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.265892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.265904 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.265936 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:11:34.266732 (+ 796us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.267535 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.269084 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.269402 (+ 318us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.269441 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.272444 (+ 3003us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:11:34.272446 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:11:34.272446 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:11:34.272453 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:11:34.274018 (+ 1565us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.274492 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.274493 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.274493 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.274557 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.274973 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.274974 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.274975 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.275148 (+ 173us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:11:34.275193 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.275567 (+ 374us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.275666 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:11:34.276304 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.264910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45281 (local address 127.16.122.131:52135)
0923 23:11:34.265074 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.265125 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.265768 (+ 643us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.266114 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.266118 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.266354 (+ 236us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:11:34.266631 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.266636 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.267668 (+ 1032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.267669 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.268985 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.268988 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.269534 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.269535 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.269688 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.270213 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:11:34.270223 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:11:34.272339 (+ 2116us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:11:34.274110 (+ 1771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:34.274118 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:34.274121 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:34.274408 (+ 287us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:34.274732 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:11:34.274733 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:11:34.274733 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:11:34.274892 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:11:34.275237 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:11:34.275243 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:11:34.275409 (+ 166us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.275902 (+ 493us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.276066 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":87,"threads_started":1}
I20260923 23:11:34.276714 28558 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45281
I20260923 23:11:34.276808 28558 heartbeater.cc:464] Registering TS with master...
I20260923 23:11:34.276995 28558 heartbeater.cc:511] Master 127.16.122.190:45281 requested a full tablet report, sending...
I20260923 23:11:34.277316 28074 ts_manager.cc:194] Registered new tserver with Master: 248b0fbfb6504f53830930c33b923dc0 (127.16.122.131:36567)
I20260923 23:11:34.277832 28074 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:52135
I20260923 23:11:34.281850 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:11:34.294778 28288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.286904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49852 (local address 127.16.122.190:45281)
0923 23:11:34.286944 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.286946 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.287284 (+ 338us) server_negotiation.cc:407] Connection header received
0923 23:11:34.287379 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.287380 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.287390 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.287418 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:11:34.288169 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.288972 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.290083 (+ 1111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.290377 (+ 294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.290416 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.291981 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:11:34.291983 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:11:34.291983 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:11:34.291990 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:11:34.293577 (+ 1587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.293968 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.293969 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.293969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.294020 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:11:34.294200 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:11:34.294201 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:11:34.294201 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:11:34.294319 (+ 118us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:11:34.294388 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.294590 (+ 202us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.294679 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:11:34.296706 28074 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:11:34.298919 28074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:11:34.312331 28573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41833 (local address 127.0.0.1:59822)
0923 23:11:34.303393 (+ 1373us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.303434 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.303500 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.303823 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.303826 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.303829 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:11:34.303892 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.303894 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.305605 (+ 1711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.305606 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.306938 (+ 1332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.306941 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.311823 (+ 4882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.311824 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.311956 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.311978 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.312045 (+ 67us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.312100 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1346,"thread_start_us":73,"threads_started":1}
I20260923 23:11:34.312682 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302119 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59822 (local address 127.16.122.130:41833)
0923 23:11:34.303333 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.303338 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.303578 (+ 240us) server_negotiation.cc:407] Connection header received
0923 23:11:34.303622 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.303625 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.303640 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.303744 (+ 104us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:11:34.303973 (+ 229us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.305520 (+ 1547us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.307850 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.311641 (+ 3791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.311732 (+ 91us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.311791 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.312549 (+ 758us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.312594 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1176,"thread_start_us":86,"threads_started":1}
I20260923 23:11:34.313421 28357 tablet_service.cc:1467] Processing CreateTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 (DEFAULT_TABLE table=test-table [id=b9f10d6ed7094336a9642adc4dbc67b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:11:34.313783 28357 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871e1412ca3c4bb4a8ec5535962dad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:11:34.314282 28575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:36567 (local address 127.0.0.1:53292)
0923 23:11:34.307317 (+ 4805us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.307326 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.307362 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.307717 (+ 355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.307718 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.307721 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:11:34.307760 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.307762 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.309517 (+ 1755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.309518 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.310993 (+ 1475us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.310995 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.314068 (+ 3073us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.314069 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.314127 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.314131 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.314177 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.314216 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4790,"thread_start_us":52,"threads_started":1}
I20260923 23:11:34.314543 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53292 (local address 127.16.122.131:36567)
0923 23:11:34.307485 (+ 4742us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.307491 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.307500 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:11:34.307536 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.307538 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.307554 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.307621 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:11:34.308277 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.309422 (+ 1145us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.312845 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.313902 (+ 1057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.313984 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.314035 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.314429 (+ 394us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.314470 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4697,"thread_start_us":67,"threads_started":1}
I20260923 23:11:34.315150 28492 tablet_service.cc:1467] Processing CreateTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 (DEFAULT_TABLE table=test-table [id=b9f10d6ed7094336a9642adc4dbc67b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:11:34.315460 28492 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871e1412ca3c4bb4a8ec5535962dad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:11:34.318988 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302247 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41961 (local address 127.0.0.1:38850)
0923 23:11:34.305048 (+ 2801us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.305058 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.305097 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.315339 (+ 10242us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.315340 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.315343 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:11:34.315383 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.315385 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.316693 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.316694 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.317832 (+ 1138us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.317835 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.318778 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.318780 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.318843 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.318847 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.318878 (+ 31us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.318915 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2785,"thread_start_us":66,"threads_started":1}
I20260923 23:11:34.319100 28577 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Bootstrap starting.
I20260923 23:11:34.319444 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.302352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38850 (local address 127.16.122.129:41961)
0923 23:11:34.315131 (+ 12779us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.315136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.315159 (+ 23us) server_negotiation.cc:407] Connection header received
0923 23:11:34.315196 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.315198 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.315215 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.315457 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:11:34.315524 (+ 67us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.316584 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.317934 (+ 1350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.318680 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.319223 (+ 543us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.319293 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.319321 (+ 28us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.319364 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12734,"thread_start_us":56,"threads_started":1}
I20260923 23:11:34.320138 28220 tablet_service.cc:1467] Processing CreateTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 (DEFAULT_TABLE table=test-table [id=b9f10d6ed7094336a9642adc4dbc67b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:11:34.320277 28577 tablet_bootstrap.cc:654] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Neither blocks nor log segments found. Creating new log.
I20260923 23:11:34.320492 28220 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 871e1412ca3c4bb4a8ec5535962dad10. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:11:34.320569 28577 log.cc:824] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:11:34.320968 28578 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Bootstrap starting.
I20260923 23:11:34.321846 28578 tablet_bootstrap.cc:654] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Neither blocks nor log segments found. Creating new log.
I20260923 23:11:34.322209 28578 log.cc:824] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Log is configured to *not* fsync() on all Append() calls
I20260923 23:11:34.328015 28577 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: No bootstrap required, opened a new log
I20260923 23:11:34.328111 28577 ts_tablet_manager.cc:1397] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.000s
I20260923 23:11:34.330192 28577 raft_consensus.cc:348] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.330390 28577 raft_consensus.cc:374] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:11:34.330438 28577 raft_consensus.cc:729] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ba91e992d1674b229a3a314fe5b6789f, State: Initialized, Role: FOLLOWER
I20260923 23:11:34.330561 28577 consensus_queue.cc:260] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.330868 28578 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: No bootstrap required, opened a new log
I20260923 23:11:34.330899 28577 ts_tablet_manager.cc:1428] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:11:34.330947 28578 ts_tablet_manager.cc:1397] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.000s
I20260923 23:11:34.331056 28423 heartbeater.cc:503] Master 127.16.122.190:45281 was elected leader, sending a full tablet report...
I20260923 23:11:34.331174 28581 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: Bootstrap starting.
I20260923 23:11:34.331941 28581 tablet_bootstrap.cc:654] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: Neither blocks nor log segments found. Creating new log.
I20260923 23:11:34.332242 28581 log.cc:824] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: Log is configured to *not* fsync() on all Append() calls
I20260923 23:11:34.332988 28578 raft_consensus.cc:348] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.333180 28578 raft_consensus.cc:374] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:11:34.333228 28578 raft_consensus.cc:729] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 248b0fbfb6504f53830930c33b923dc0, State: Initialized, Role: FOLLOWER
I20260923 23:11:34.333355 28578 consensus_queue.cc:260] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.333628 28578 ts_tablet_manager.cc:1428] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:11:34.333751 28558 heartbeater.cc:503] Master 127.16.122.190:45281 was elected leader, sending a full tablet report...
I20260923 23:11:34.335578 28581 tablet_bootstrap.cc:492] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: No bootstrap required, opened a new log
I20260923 23:11:34.335659 28581 ts_tablet_manager.cc:1397] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.000s
I20260923 23:11:34.338724 28581 raft_consensus.cc:348] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.338927 28581 raft_consensus.cc:374] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:11:34.338960 28581 raft_consensus.cc:729] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 43cb6c4b2db94eb2814b90928cedcbf4, State: Initialized, Role: FOLLOWER
I20260923 23:11:34.339692 28581 consensus_queue.cc:260] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.340026 28581 ts_tablet_manager.cc:1428] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4: Time spent starting tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:11:34.340144 28286 heartbeater.cc:503] Master 127.16.122.190:45281 was elected leader, sending a full tablet report...
I20260923 23:11:34.341622 28584 raft_consensus.cc:482] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:11:34.341737 28584 raft_consensus.cc:504] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.342078 28584 leader_election.cc:288] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ba91e992d1674b229a3a314fe5b6789f (127.16.122.130:41833), 43cb6c4b2db94eb2814b90928cedcbf4 (127.16.122.129:41961)
I20260923 23:11:34.346411 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.342302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:41833 (local address 127.16.122.131:40061)
0923 23:11:34.342587 (+ 285us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.342599 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.342640 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.343010 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.343012 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.343013 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:11:34.343117 (+ 104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.343119 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.344069 (+ 950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.344070 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.345299 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.345302 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.346138 (+ 836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.346139 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.346240 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.346244 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.346290 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.346334 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266}
I20260923 23:11:34.346817 28572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.342768 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40061 (local address 127.16.122.130:41833)
0923 23:11:34.342869 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.342871 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.342878 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:11:34.342902 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.342902 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.342913 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.342961 (+ 48us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:11:34.343203 (+ 242us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.343957 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.345396 (+ 1439us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.345992 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.346053 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.346061 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.346706 (+ 645us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.346751 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260923 23:11:34.347126 28377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871e1412ca3c4bb4a8ec5535962dad10" candidate_uuid: "248b0fbfb6504f53830930c33b923dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba91e992d1674b229a3a314fe5b6789f" is_pre_election: true
I20260923 23:11:34.347319 28377 raft_consensus.cc:2454] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248b0fbfb6504f53830930c33b923dc0 in term 0.
I20260923 23:11:34.347546 28445 leader_election.cc:302] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248b0fbfb6504f53830930c33b923dc0, ba91e992d1674b229a3a314fe5b6789f; no voters:
I20260923 23:11:34.347690 28584 raft_consensus.cc:2781] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:11:34.347738 28584 raft_consensus.cc:482] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:11:34.347760 28584 raft_consensus.cc:3037] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:11:34.348695 28584 raft_consensus.cc:504] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.348829 28584 leader_election.cc:288] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [CANDIDATE]: Term 1 election: Requested vote from peers ba91e992d1674b229a3a314fe5b6789f (127.16.122.130:41833), 43cb6c4b2db94eb2814b90928cedcbf4 (127.16.122.129:41961)
I20260923 23:11:34.349026 28377 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871e1412ca3c4bb4a8ec5535962dad10" candidate_uuid: "248b0fbfb6504f53830930c33b923dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ba91e992d1674b229a3a314fe5b6789f"
I20260923 23:11:34.349099 28377 raft_consensus.cc:3037] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:11:34.349972 28377 raft_consensus.cc:2454] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248b0fbfb6504f53830930c33b923dc0 in term 1.
I20260923 23:11:34.350170 28445 leader_election.cc:302] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 248b0fbfb6504f53830930c33b923dc0, ba91e992d1674b229a3a314fe5b6789f; no voters:
I20260923 23:11:34.350247 28584 raft_consensus.cc:2781] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:11:34.350394 28584 raft_consensus.cc:686] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 1 LEADER]: Becoming Leader. State: Replica: 248b0fbfb6504f53830930c33b923dc0, State: Running, Role: LEADER
I20260923 23:11:34.350497 28584 consensus_queue.cc:237] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } }
I20260923 23:11:34.350749 28574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.342822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:60717 (local address 127.16.122.129:41961)
0923 23:11:34.342875 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.342877 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.346678 (+ 3801us) server_negotiation.cc:407] Connection header received
0923 23:11:34.346708 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.346709 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.346719 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.346785 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:11:34.347104 (+ 319us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.347795 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.349459 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.350040 (+ 581us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.350107 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.350116 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.350636 (+ 520us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.350684 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:11:34.351050 28560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.342392 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41961 (local address 127.16.122.131:60717)
0923 23:11:34.346494 (+ 4102us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.346505 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:11:34.346546 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:11:34.346833 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:11:34.346834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:11:34.346836 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:11:34.346870 (+ 34us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.346872 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.348057 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.348058 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.349178 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:11:34.349180 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.350340 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.350341 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:11:34.350405 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.350409 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:11:34.350524 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 23:11:34.350565 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4089,"thread_start_us":93,"threads_started":1}
I20260923 23:11:34.351400 28074 catalog_manager.cc:5495] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 reported cstate change: term changed from 0 to 1, leader changed from <none> to 248b0fbfb6504f53830930c33b923dc0 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "248b0fbfb6504f53830930c33b923dc0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "248b0fbfb6504f53830930c33b923dc0" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 36567 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 } health_report { overall_health: UNKNOWN } } }
I20260923 23:11:34.352687 28240 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871e1412ca3c4bb4a8ec5535962dad10" candidate_uuid: "248b0fbfb6504f53830930c33b923dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" is_pre_election: true
I20260923 23:11:34.352859 28240 raft_consensus.cc:2454] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 248b0fbfb6504f53830930c33b923dc0 in term 0.
I20260923 23:11:34.352900 28239 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "871e1412ca3c4bb4a8ec5535962dad10" candidate_uuid: "248b0fbfb6504f53830930c33b923dc0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "43cb6c4b2db94eb2814b90928cedcbf4"
I20260923 23:11:34.352965 28239 raft_consensus.cc:3037] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:11:34.353796 28239 raft_consensus.cc:2454] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 248b0fbfb6504f53830930c33b923dc0 in term 1.
I20260923 23:11:34.372916 28576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.369298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53296 (local address 127.16.122.131:36567)
0923 23:11:34.369339 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:11:34.369341 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:11:34.369348 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:11:34.369393 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:11:34.369393 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:11:34.369415 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:11:34.369479 (+ 64us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:11:34.369726 (+ 247us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.370455 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.371623 (+ 1168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:11:34.371950 (+ 327us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:11:34.372032 (+ 82us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:11:34.372298 (+ 266us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:11:34.372445 (+ 147us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:11:34.372523 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:11:34.372784 (+ 261us) server_negotiation.cc:299] Negotiation successful
0923 23:11:34.372807 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:11:34.380300 28239 raft_consensus.cc:1261] T 871e1412ca3c4bb4a8ec5535962dad10 P 43cb6c4b2db94eb2814b90928cedcbf4 [term 1 FOLLOWER]: Refusing update from remote peer 248b0fbfb6504f53830930c33b923dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:11:34.380679 28377 raft_consensus.cc:1261] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 1 FOLLOWER]: Refusing update from remote peer 248b0fbfb6504f53830930c33b923dc0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:11:34.381110 28584 consensus_queue.cc:1035] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ba91e992d1674b229a3a314fe5b6789f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 41833 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:11:34.381294 28584 consensus_queue.cc:1035] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "43cb6c4b2db94eb2814b90928cedcbf4" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 41961 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:11:34.388711 28591 mvcc.cc:205] Tried to move back new op lower bound from 7332680066577940480 to 7332680066460381184. Current Snapshot: MvccSnapshot[applied={T|T < 7332680066577940480}]
I20260923 23:11:34.395942 28074 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:11:34.396063 28074 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:11:34.396790 28074 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cab18a7dc4164a5fbb88e25f4323af14: Sending DeleteTablet for 3 replicas of tablet 871e1412ca3c4bb4a8ec5535962dad10
I20260923 23:11:34.397229 28357 tablet_service.cc:1514] Processing DeleteTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:11:34 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:59822
I20260923 23:11:34.397475 28220 tablet_service.cc:1514] Processing DeleteTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:11:34 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38850
I20260923 23:11:34.397862 28492 tablet_service.cc:1514] Processing DeleteTablet for tablet 871e1412ca3c4bb4a8ec5535962dad10 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:11:34 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:53292
I20260923 23:11:34.398438 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 28155
I20260923 23:11:34.399252 28599 tablet_replica.cc:330] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: stopping tablet replica
I20260923 23:11:34.399439 28599 raft_consensus.cc:2229] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:11:34.399549 28599 raft_consensus.cc:2258] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:11:34.400004 28599 ts_tablet_manager.cc:1905] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:11:34.403154 28599 ts_tablet_manager.cc:1918] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:11:34.403232 28599 log.cc:1196] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/871e1412ca3c4bb4a8ec5535962dad10
I20260923 23:11:34.403398 28601 tablet_replica.cc:330] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: stopping tablet replica
I20260923 23:11:34.403563 28601 raft_consensus.cc:2229] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:11:34.403759 28601 raft_consensus.cc:2258] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:11:34.403788 28599 ts_tablet_manager.cc:1939] T 871e1412ca3c4bb4a8ec5535962dad10 P ba91e992d1674b229a3a314fe5b6789f: Deleting consensus metadata
I20260923 23:11:34.404202 28058 catalog_manager.cc:4841] TS ba91e992d1674b229a3a314fe5b6789f (127.16.122.130:41833): tablet 871e1412ca3c4bb4a8ec5535962dad10 (table test-table [id=b9f10d6ed7094336a9642adc4dbc67b7]) successfully deleted
I20260923 23:11:34.404358 28601 ts_tablet_manager.cc:1905] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:11:34.405887 28601 ts_tablet_manager.cc:1918] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:11:34.407387 28601 log.cc:1196] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/871e1412ca3c4bb4a8ec5535962dad10
I20260923 23:11:34.407714 28601 ts_tablet_manager.cc:1939] T 871e1412ca3c4bb4a8ec5535962dad10 P 248b0fbfb6504f53830930c33b923dc0: Deleting consensus metadata
I20260923 23:11:34.408195 28061 catalog_manager.cc:4841] TS 248b0fbfb6504f53830930c33b923dc0 (127.16.122.131:36567): tablet 871e1412ca3c4bb4a8ec5535962dad10 (table test-table [id=b9f10d6ed7094336a9642adc4dbc67b7]) successfully deleted
W20260923 23:11:34.408550 28058 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:41961 (error 108)
I20260923 23:11:34.409148 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 28292
I20260923 23:11:34.414444 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 28429
I20260923 23:11:34.415395 28571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:11:34.415281 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:41961 (local address 127.0.0.1:38856)
0923 23:11:34.415327 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:11:34.415360 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:41961: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:11:34.415599 28058 catalog_manager.cc:4584] TS 43cb6c4b2db94eb2814b90928cedcbf4 (127.16.122.129:41961): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 871e1412ca3c4bb4a8ec5535962dad10: Network error: Client connection negotiation failed: client connection to 127.16.122.129:41961: connect: Connection refused (error 111)
I20260923 23:11:34.424329 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 28042
I20260923 23:11:34.430083 16874 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 23:11:34.764207 16874 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 23:11:34.764317 16874 mini_ranger.cc:66] Stopping Ranger...
I20260923 23:11:35.098527 16874 mini_ranger.cc:68] Stopped Ranger
2026-09-23 23:11:35.098 UTC [27628] LOG: received smart shutdown request
2026-09-23 23:11:35.103 UTC [27628] LOG: background worker "logical replication launcher" (PID 27637) exited with exit code 1
2026-09-23 23:11:35.103 UTC [27632] LOG: shutting down
2026-09-23 23:11:35.256 UTC [27628] LOG: database system is shut down
2026-09-23T23:11:35Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (43450 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:11:35 dist-test-slave-95hh krb5kdc[28611](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:11:35 dist-test-slave-95hh krb5kdc[28611](info): set up 2 sockets
Sep 23 23:11:35 dist-test-slave-95hh krb5kdc[28611](info): commencing operation
krb5kdc: starting...
W20260923 23:11:36.294669 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.004s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:11:36 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:11:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:11:36Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangeradmin_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangeradmin_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangerlookup_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangerlookup_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
I20260923 23:11:36.443337 16874 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskODLJMS/build/release/bin/postgres/pg_ctl -D /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/postgres -l logfile start
2026-09-23 23:11:38.661 UTC [28641] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-09-23 23:11:38.662 UTC [28641] LOG: listening on IPv4 address "127.16.122.148", port 37197
2026-09-23 23:11:38.665 UTC [28641] LOG: listening on Unix socket "/tmp/.s.PGSQL.37197"
2026-09-23 23:11:38.691 UTC [28644] LOG: database system was shut down at 2026-09-23 23:11:37 UTC
2026-09-23 23:11:38.695 UTC [28641] LOG: database system is ready to accept connections
I20260923 23:11:39.566685 16874 mini_postgres.cc:96] Postgres bound to 37197
2026-09-23 23:11:39.572 UTC [28652] FATAL: database "slave" does not exist
127.16.122.148:37197 - accepting connections
I20260923 23:11:39.572660 16874 mini_ranger.cc:161] Starting Ranger...
I20260923 23:11:39.586330 16874 mini_ranger.cc:84] Created miniranger Postgres user
I20260923 23:11:39.994148 16874 mini_ranger.cc:87] Created ranger Postgres database
I20260923 23:11:39.994216 16874 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin
2026-09-23 23:11:40,253 [I] DB FLAVOR :POSTGRES
2026-09-23 23:11:40,253 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:11:40,253 [I] Checking connection..
2026-09-23 23:11:40,254 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-09-23 23:11:40,501 [I] Checking connection passed.
2026-09-23 23:11:40,501 [I] --------- Verifying version history table ---------
2026-09-23 23:11:40,501 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:11:40,728 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:11:40,729 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:11:40,962 [I] Table x_db_version_h does not exist in database ranger
2026-09-23 23:11:40,962 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-09-23 23:11:40,963 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-09-23 23:11:41.194 UTC [28716] WARNING: there is no transaction in progress
2026-09-23 23:11:41,199 [I] create_dbversion_catalog.sql file imported successfully
2026-09-23 23:11:41,199 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-09-23 23:11:41,443 [I] Table x_db_version_h already exists in database 'ranger'
2026-09-23 23:11:41,444 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:11:41,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-09-23 23:11:41,678 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-09-23 23:11:41,928 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-95hh.c.gcp-upstream.internal','N') ;"
2026-09-23 23:11:42,171 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:11:42,421 [I] Table x_portal_user does not exist in database ranger
2026-09-23 23:11:42,422 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:11:42,676 [I] Table x_policy_ref_group does not exist in database ranger
2026-09-23 23:11:42,676 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:11:42,925 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:11:43,164 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-09-23 23:11:43,164 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-09-23 23:11:44.072 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.088 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.104 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.119 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.134 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.151 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.205 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.212 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.226 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.236 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.246 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.259 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.269 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.275 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.281 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44.288 UTC [28824] WARNING: there is no transaction in progress
2026-09-23 23:11:44,892 [I] ranger_core_db_postgres.sql file imported successfully
2026-09-23 23:11:44,892 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-09-23 23:11:45,139 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-09-23 23:11:45,139 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-09-23 23:11:45,396 [I] Table x_portal_user already exists in database 'ranger'
2026-09-23 23:11:45,396 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-09-23 23:11:45,634 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-09-23 23:11:45,634 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-09-23 23:11:45,883 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-09-23 23:11:46,110 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-95hh.c.gcp-upstream.internal';"
2026-09-23 23:11:46,341 [I] CORE_DB_SCHEMA import status has been updated
2026-09-23 23:11:46,342 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-09-23 23:11:46,590 [I] DB_PATCHES have already been applied
I20260923 23:11:46.595552 16874 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskODLJMS/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260923 23:11:46.595613 16874 mini_ranger.cc:193] Using host: 127.16.122.148
I20260923 23:11:46.598390 16874 mini_ranger.cc:227] Ranger admin URL: http://127.16.122.148:48781
Sep 23, 2026 11:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:11:46 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp
Sep 23, 2026 11:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Sep 23, 2026 11:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp] .....
Sep 23, 2026 11:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-admin/ews/webapp].
Sep 23, 2026 11:11:47 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260923 23:11:47.596073 16874 mini_ranger.cc:160] Time spent starting Ranger: real 8.023s user 0.000s sys 0.007s
Sep 23, 2026 11:12:06 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Sep 23, 2026 11:12:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.UserREST
Sep 23, 2026 11:12:07 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Sep 23, 2026 11:12:07 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Sep 23, 2026 11:12:07 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260923 23:12:08.518787 16874 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.122.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.122.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangerkms_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.122.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/rangerkms_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.122.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.122.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.122.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/HTTP_127.16.122.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Sep 23 23:12:08 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205128, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260923 23:12:08.599232 16874 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260923 23:12:08.608475 16874 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260923 23:12:09.483539 16874 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260923 23:12:09.483628 16874 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms
I20260923 23:12:09.483645 16874 mini_ranger_kms.cc:227] Using postgres at 127.16.122.148:37197
2026-09-23 23:12:09,629 [I] DB FLAVOR :POSTGRES
2026-09-23 23:12:09,629 [I] --------- Verifying Ranger DB connection ---------
2026-09-23 23:12:09,630 [I] Checking connection
2026-09-23 23:12:09,630 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-09-23 23:12:09,870 [I] connection success
2026-09-23 23:12:09,870 [I] --------- Verifying Ranger DB tables ---------
2026-09-23 23:12:09,871 [I] Verifying table ranger_masterkey in database rangerkms
2026-09-23 23:12:09,871 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-09-23 23:12:10,127 [I] Table ranger_masterkey does not exist in database rangerkms
2026-09-23 23:12:10,127 [I] --------- Importing Ranger Core DB Schema ---------
2026-09-23 23:12:10,127 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-09-23 23:12:10,127 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.122.148:37197/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-09-23 23:12:10,406 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260923 23:12:10.900892 16874 mini_ranger_kms.cc:310] Created kms service
I20260923 23:12:11.118584 16874 mini_ranger_kms.cc:326] Created kudu user
I20260923 23:12:11.170853 16874 mini_ranger_kms.cc:343] Created rangerkms user
Sep 23 23:12:11 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205128, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:11.535758 16874 mini_ranger_kms.cc:384] Added ranger policy
I20260923 23:12:11.535877 16874 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms:/tmp/dist-test-taskODLJMS/build/release/bin/postgresql.jar:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskODLJMS/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260923 23:12:11.535894 16874 mini_ranger_kms.cc:240] Using host: 127.16.122.148
I20260923 23:12:11.539079 16874 mini_ranger_kms.cc:276] Ranger KMS PID: 29004
I20260923 23:12:11.539167 16874 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.122.148:37749
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp] .....
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-kms/ews/webapp].
Sep 23, 2026 11:12:12 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
I20260923 23:12:13.598841 16874 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 5.000s user 0.000s sys 0.130s
I20260923 23:12:13.598950 16874 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260923 23:12:13.598999 16874 mini_ranger_kms.cc:401] 127.16.122.148:37749/kms/v1/keys
Sep 23 23:12:13 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790205133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 23 23:12:14 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
Sep 23 23:12:14 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205133, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.122.148@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
Sep 23 23:12:16 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:12:16.589408 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42213
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:42213
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:16.667711 29055 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:16.667937 29055 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:16.667968 29055 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:16.669132 29055 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:16.669174 29055 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:16.669193 29055 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:16.669207 29055 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:16.669224 29055 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:16.670744 29055 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.122.148:37749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:42213
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29055
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:16.670982 29055 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:16.671200 29055 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:16.676151 29063 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:16.676158 29060 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:16.676321 29055 server_base.cc:1034] running on GCE node
W20260923 23:12:16.676386 29061 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:16.676752 29055 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:16.677045 29055 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:16.678187 29055 hybrid_clock.cc:648] HybridClock initialized: now 1790205136678172 us; error 32 us; skew 500 ppm
Sep 23 23:12:16 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:16.684928 29055 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:16.685425 29055 webserver.cc:460] Webserver started at http://127.16.122.190:34775/ using document root <none> and password file <none>
I20260923 23:12:16.685624 29055 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:16.685665 29055 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:16.685789 29055 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:12:16 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205136, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:16.734930 29055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "70459954dad94901ac445c7272a9308d"
format_stamp: "Formatted at 2026-09-23 23:12:16 on dist-test-slave-95hh"
server_key: "58d0bf52efb466277ace356a83382088"
server_key_iv: "339ed29e724abcd332e365150447e445"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:16.735576 29055 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "70459954dad94901ac445c7272a9308d"
format_stamp: "Formatted at 2026-09-23 23:12:16 on dist-test-slave-95hh"
server_key: "58d0bf52efb466277ace356a83382088"
server_key_iv: "339ed29e724abcd332e365150447e445"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:16.737076 29055 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:16.750782 29070 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:16.751013 29055 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:16.751134 29055 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "70459954dad94901ac445c7272a9308d"
format_stamp: "Formatted at 2026-09-23 23:12:16 on dist-test-slave-95hh"
server_key: "58d0bf52efb466277ace356a83382088"
server_key_iv: "339ed29e724abcd332e365150447e445"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:16.751226 29055 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:16.759280 29055 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:16.761090 29055 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:16.761211 29055 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:16.765585 29055 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:42213
I20260923 23:12:16.765978 29055 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:16.766333 29122 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:42213 every 8 connection(s)
I20260923 23:12:16.766681 29123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:16.768951 29123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Bootstrap starting.
I20260923 23:12:16.769518 29123 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:16.769771 29123 log.cc:824] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:16.770394 29123 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: No bootstrap required, opened a new log
I20260923 23:12:16.772372 29123 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:16.772521 29123 raft_consensus.cc:374] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:16.772542 29123 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70459954dad94901ac445c7272a9308d, State: Initialized, Role: FOLLOWER
I20260923 23:12:16.772666 29123 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:16.772734 29123 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:16.772766 29123 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:16.772814 29123 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:16.773301 29123 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:16.773392 29123 leader_election.cc:302] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70459954dad94901ac445c7272a9308d; no voters:
I20260923 23:12:16.773538 29123 leader_election.cc:288] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:12:16.773581 29126 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:16.773736 29126 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 LEADER]: Becoming Leader. State: Replica: 70459954dad94901ac445c7272a9308d, State: Running, Role: LEADER
I20260923 23:12:16.773849 29123 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:16.773873 29126 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:16.774282 29126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70459954dad94901ac445c7272a9308d. Latest consensus state: current_term: 1 leader_uuid: "70459954dad94901ac445c7272a9308d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } } }
I20260923 23:12:16.774506 29126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:16.774444 29127 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70459954dad94901ac445c7272a9308d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } } }
I20260923 23:12:16.774859 29127 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:16.775053 29135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:16.775166 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29055
I20260923 23:12:16.775203 29123 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 23:12:16.775269 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:12:16.775439 16874 external_mini_cluster.cc:1428] Setting key 72fa9578c59e4c0d50e41f40a9120aa2
I20260923 23:12:16.775599 29135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:16.781209 29135 catalog_manager.cc:1347] Generated new cluster ID: 7d4543b93eb94aafa638bab56abda86d
I20260923 23:12:16.781345 29135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:12:16 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:16.791401 29145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:16.775900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56212 (local address 127.16.122.190:42213)
0923 23:12:16.779743 (+ 3843us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:16.779752 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:16.779793 (+ 41us) server_negotiation.cc:407] Connection header received
0923 23:12:16.780011 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:16.780022 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:16.780102 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:16.780281 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:16.781044 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:16.782621 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:16.783757 (+ 1136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:16.784374 (+ 617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:16.784476 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:16.787671 (+ 3195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:16.787677 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:16.787678 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:16.787691 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:16.789499 (+ 1808us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:16.789857 (+ 358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:16.789861 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:16.789862 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:16.789960 (+ 98us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:16.790118 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:16.790118 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:16.790119 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:16.790423 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:16.790549 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:16.791004 (+ 455us) server_negotiation.cc:299] Negotiation successful
0923 23:12:16.791163 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3734,"thread_start_us":77,"threads_started":1}
I20260923 23:12:16.827864 29135 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:12:16.828691 29135 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:16.832963 29135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Generated new TSK 0
I20260923 23:12:16.833209 29135 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:12:17.526417 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:17.634119 29168 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:17.634351 29168 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:17.634390 29168 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:17.635620 29168 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:17.635663 29168 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:17.635705 29168 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:17.637163 29168 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29168
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:17.637424 29168 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:17.637636 29168 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:17.643669 29173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:17.643677 29174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:17.643909 29176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:17.643937 29168 server_base.cc:1034] running on GCE node
I20260923 23:12:17.644132 29168 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:17.644388 29168 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:17.645543 29168 hybrid_clock.cc:648] HybridClock initialized: now 1790205137645528 us; error 31 us; skew 500 ppm
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:17.648029 29168 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:17.648478 29168 webserver.cc:460] Webserver started at http://127.16.122.129:39879/ using document root <none> and password file <none>
I20260923 23:12:17.648689 29168 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:17.648736 29168 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:17.648865 29168 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:17.661355 29168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "3a8c386bffe0d2251f34b4426a5377ab"
server_key_iv: "eacb86b43c010c789799113feed8aab9"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.661791 29168 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "3a8c386bffe0d2251f34b4426a5377ab"
server_key_iv: "eacb86b43c010c789799113feed8aab9"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.663565 29168 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.003s sys 0.000s
I20260923 23:12:17.681859 29183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.682094 29168 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:17.682276 29168 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "3a8c386bffe0d2251f34b4426a5377ab"
server_key_iv: "eacb86b43c010c789799113feed8aab9"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.682446 29168 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:17.704207 29168 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:17.706705 29168 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:17.706915 29168 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:17.707232 29168 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:17.707695 29168 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:17.707751 29168 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.707780 29168 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:17.707836 29168 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.715569 29168 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:34137
I20260923 23:12:17.715911 29168 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:17.716434 29298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:34137 every 8 connection(s)
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:17.722851 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29168
I20260923 23:12:17.722944 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:12:17.723232 16874 external_mini_cluster.cc:1428] Setting key 10a61241d5caf80f351e9e6840795d81
I20260923 23:12:17.726924 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:17.717044 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:35607 (local address 127.16.122.190:42213)
0923 23:12:17.717173 (+ 129us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:17.717175 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:17.717302 (+ 127us) server_negotiation.cc:407] Connection header received
0923 23:12:17.717792 (+ 490us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:17.717793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:17.717806 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:17.717878 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:17.718503 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.719305 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.720274 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.720566 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.720637 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:17.723540 (+ 2903us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:17.723541 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:17.723542 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:17.723550 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:17.725375 (+ 1825us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:17.725839 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:17.725840 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:17.725840 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:17.725911 (+ 71us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:17.726196 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:17.726198 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:17.726198 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:17.726348 (+ 150us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:17.726425 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:17.726735 (+ 310us) server_negotiation.cc:299] Negotiation successful
0923 23:12:17.726825 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":72,"threads_started":1}
I20260923 23:12:17.727092 29301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:17.717017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.129:35607)
0923 23:12:17.717182 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:17.717226 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:17.717696 (+ 470us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:17.717948 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:17.717951 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:17.718125 (+ 174us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:17.718388 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:17.718395 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.719409 (+ 1014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.719410 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:17.720183 (+ 773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:17.720185 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.720680 (+ 495us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.720681 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:17.720817 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:17.721408 (+ 591us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:17.721418 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:17.723334 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:17.725463 (+ 2129us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:17.725470 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:17.725472 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:17.725748 (+ 276us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:17.725985 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:17.725986 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:17.725986 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:17.726115 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:17.726419 (+ 304us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:17.726424 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:17.726581 (+ 157us) client_negotiation.cc:769] Sending connection context
0923 23:12:17.726687 (+ 106us) client_negotiation.cc:240] Negotiation successful
0923 23:12:17.726833 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":81,"threads_started":1}
I20260923 23:12:17.728130 29299 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:17.728227 29299 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:17.728415 29299 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:17.728971 29087 ts_manager.cc:194] Registered new tserver with Master: b2afc23ab05545a1a23f6cfcdd6a6efb (127.16.122.129:34137)
I20260923 23:12:17.729835 29087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:35607
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:12:17.757640 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260923 23:12:17.871990 29305 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:17.872318 29305 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:17.872368 29305 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:17.874287 29305 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:17.874344 29305 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:17.874385 29305 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:17.876704 29305 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29305
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:17.877089 29305 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:17.877393 29305 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:17.883828 29315 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:17.884786 29305 server_base.cc:1034] running on GCE node
W20260923 23:12:17.884999 29312 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:17.885154 29313 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:17.885403 29305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:17.885679 29305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:17.886830 29305 hybrid_clock.cc:648] HybridClock initialized: now 1790205137886808 us; error 34 us; skew 500 ppm
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:17.890275 29305 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:17.890813 29305 webserver.cc:460] Webserver started at http://127.16.122.130:41905/ using document root <none> and password file <none>
I20260923 23:12:17.891113 29305 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:17.891168 29305 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:17.891305 29305 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:17.906988 29305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "6e0f808d43184b7fbda3613b60c8afa7"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "69353996a4c23024988fee47817f922c"
server_key_iv: "f9113debe8f21f4015ab0c664444fa94"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.907591 29305 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "6e0f808d43184b7fbda3613b60c8afa7"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "69353996a4c23024988fee47817f922c"
server_key_iv: "f9113debe8f21f4015ab0c664444fa94"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.909410 29305 fs_manager.cc:691] Time spent creating directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:17.923445 29322 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.935105 29305 fs_manager.cc:725] Time spent opening block manager: real 0.013s user 0.001s sys 0.000s
I20260923 23:12:17.935245 29305 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "6e0f808d43184b7fbda3613b60c8afa7"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "69353996a4c23024988fee47817f922c"
server_key_iv: "f9113debe8f21f4015ab0c664444fa94"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:17.935346 29305 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:17.963979 29305 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:17.966317 29305 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:17.966511 29305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:17.966809 29305 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:17.967254 29305 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:17.967343 29305 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.967413 29305 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:17.967473 29305 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:17.973407 29305 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:33369
I20260923 23:12:17.973909 29305 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:12:17.975071 29436 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:33369 every 8 connection(s)
I20260923 23:12:17.984066 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29305
I20260923 23:12:17.984175 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:12:17.984380 16874 external_mini_cluster.cc:1428] Setting key 431f13bc8ee81a0eb2a5c46dab55b806
Sep 23 23:12:17 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205137, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:17.988962 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:17.980047 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:43381 (local address 127.16.122.190:42213)
0923 23:12:17.980126 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:17.980128 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:17.980329 (+ 201us) server_negotiation.cc:407] Connection header received
0923 23:12:17.980815 (+ 486us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:17.980816 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:17.980831 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:17.980877 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:17.981560 (+ 683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.982245 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.983259 (+ 1014us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.983467 (+ 208us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.983523 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:17.985703 (+ 2180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:17.985704 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:17.985704 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:17.985712 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:17.987303 (+ 1591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:17.987817 (+ 514us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:17.987819 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:17.987820 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:17.987881 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:17.988162 (+ 281us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:17.988163 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:17.988163 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:17.988314 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:17.988359 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:17.988752 (+ 393us) server_negotiation.cc:299] Negotiation successful
0923 23:12:17.988844 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:12:17.989701 29439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:17.980017 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.130:43381)
0923 23:12:17.980191 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:17.980244 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:17.980721 (+ 477us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:17.980948 (+ 227us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:17.980952 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:17.981155 (+ 203us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:17.981443 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:17.981450 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.982339 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.982340 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:17.983182 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:17.983185 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:17.983557 (+ 372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:17.983558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:17.983660 (+ 102us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:17.984150 (+ 490us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:17.984161 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:17.985624 (+ 1463us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:17.987424 (+ 1800us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:17.987431 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:17.987433 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:17.987724 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:17.987950 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:17.987951 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:17.987951 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:17.988085 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:17.988406 (+ 321us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:17.988413 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:17.988578 (+ 165us) client_negotiation.cc:769] Sending connection context
0923 23:12:17.989297 (+ 719us) client_negotiation.cc:240] Negotiation successful
0923 23:12:17.989464 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":94,"threads_started":1}
I20260923 23:12:17.990352 29437 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:17.990444 29437 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:17.990623 29437 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:17.991077 29087 ts_manager.cc:194] Registered new tserver with Master: 6e0f808d43184b7fbda3613b60c8afa7 (127.16.122.130:33369)
I20260923 23:12:17.991559 29087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:43381
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:12:18.018015 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:18.128394 29442 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:18.128612 29442 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:18.128644 29442 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:18.129809 29442 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:18.129866 29442 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:18.129909 29442 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:18.131415 29442 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29442
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:18.131652 29442 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:18.131873 29442 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:18.136639 29447 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.136662 29448 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.136655 29450 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.145582 29442 server_base.cc:1034] running on GCE node
I20260923 23:12:18.146041 29442 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:18.146358 29442 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:18.147529 29442 hybrid_clock.cc:648] HybridClock initialized: now 1790205138147505 us; error 38 us; skew 500 ppm
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:18.150888 29442 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:18.151466 29442 webserver.cc:460] Webserver started at http://127.16.122.131:40217/ using document root <none> and password file <none>
I20260923 23:12:18.151727 29442 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:18.151777 29442 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:18.151901 29442 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:18.167568 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "2f3e6565984245c880724f58a341abdf"
format_stamp: "Formatted at 2026-09-23 23:12:18 on dist-test-slave-95hh"
server_key: "eff771637ce9ade1a3ff797e69a2b46b"
server_key_iv: "944d89ed379879d0cf0857afd74d1087"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.171365 29442 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "2f3e6565984245c880724f58a341abdf"
format_stamp: "Formatted at 2026-09-23 23:12:18 on dist-test-slave-95hh"
server_key: "eff771637ce9ade1a3ff797e69a2b46b"
server_key_iv: "944d89ed379879d0cf0857afd74d1087"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.175218 29442 fs_manager.cc:691] Time spent creating directory manager: real 0.003s user 0.001s sys 0.000s
I20260923 23:12:18.203312 29457 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.203579 29442 fs_manager.cc:725] Time spent opening block manager: real 0.010s user 0.001s sys 0.000s
I20260923 23:12:18.203677 29442 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "2f3e6565984245c880724f58a341abdf"
format_stamp: "Formatted at 2026-09-23 23:12:18 on dist-test-slave-95hh"
server_key: "eff771637ce9ade1a3ff797e69a2b46b"
server_key_iv: "944d89ed379879d0cf0857afd74d1087"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.203774 29442 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:18.215103 29442 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:18.217572 29442 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:18.217774 29442 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:18.218050 29442 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:18.218453 29442 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:18.218535 29442 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.218607 29442 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:18.218664 29442 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.224665 29442 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39971
I20260923 23:12:18.225145 29442 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:18.228221 29570 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39971 every 8 connection(s)
I20260923 23:12:18.231741 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29442
I20260923 23:12:18.231844 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:12:18.232008 16874 external_mini_cluster.cc:1428] Setting key c5dd5b4956c387cb89d5535443889e41
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:18.240116 29302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.228889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:35519 (local address 127.16.122.190:42213)
0923 23:12:18.228950 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:18.228952 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:18.229154 (+ 202us) server_negotiation.cc:407] Connection header received
0923 23:12:18.229814 (+ 660us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:18.229815 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:18.229829 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:18.229857 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:18.230754 (+ 897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.231534 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.234154 (+ 2620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.234438 (+ 284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.234475 (+ 37us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.236891 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:18.236893 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:18.236893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:18.236901 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:18.238367 (+ 1466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.238845 (+ 478us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.238846 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.238847 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.238920 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.239310 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.239311 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.239312 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.239482 (+ 170us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:18.239554 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:18.239916 (+ 362us) server_negotiation.cc:299] Negotiation successful
0923 23:12:18.240017 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:12:18.240317 29573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.228874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.131:35519)
0923 23:12:18.229035 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:18.229074 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:18.229692 (+ 618us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:18.230128 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:18.230132 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:18.230378 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:18.230649 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.230655 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.232851 (+ 2196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.232851 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.234061 (+ 1210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.234064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.234571 (+ 507us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.234572 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.234706 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.235268 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:18.235278 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:18.236793 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:18.238452 (+ 1659us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.238458 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.238461 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.238710 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.239076 (+ 366us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.239077 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.239077 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.239213 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.239585 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:18.239592 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:18.239756 (+ 164us) client_negotiation.cc:769] Sending connection context
0923 23:12:18.239875 (+ 119us) client_negotiation.cc:240] Negotiation successful
0923 23:12:18.240081 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":94,"threads_started":1}
I20260923 23:12:18.240916 29571 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:18.241012 29571 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.241189 29571 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:18.241565 29087 ts_manager.cc:194] Registered new tserver with Master: 2f3e6565984245c880724f58a341abdf (127.16.122.131:39971)
I20260923 23:12:18.242035 29087 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:35519
I20260923 23:12:18.243850 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:12:18.244524 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29168
I20260923 23:12:18.250270 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29305
I20260923 23:12:18.256203 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29442
I20260923 23:12:18.262187 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29055
I20260923 23:12:18.266619 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42213
--webserver_interface=127.16.122.190
--webserver_port=34775
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:42213
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:18.344954 29574 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:18.345173 29574 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:18.345196 29574 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:18.346347 29574 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:18.346396 29574 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:18.346410 29574 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:18.346421 29574 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:18.346431 29574 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:18.348075 29574 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:42213
--ranger_config_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:42213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=34775
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29574
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:18.348304 29574 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:18.348527 29574 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:18.354256 29582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.354274 29579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.354395 29580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.354396 29574 server_base.cc:1034] running on GCE node
I20260923 23:12:18.354724 29574 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:18.354978 29574 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:18.356128 29574 hybrid_clock.cc:648] HybridClock initialized: now 1790205138356114 us; error 30 us; skew 500 ppm
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:18.358701 29574 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:18.359148 29574 webserver.cc:460] Webserver started at http://127.16.122.190:34775/ using document root <none> and password file <none>
I20260923 23:12:18.359356 29574 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:18.359411 29574 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:18.401007 29574 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:18.401755 29590 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.401911 29574 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:18.401993 29574 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "70459954dad94901ac445c7272a9308d"
format_stamp: "Formatted at 2026-09-23 23:12:16 on dist-test-slave-95hh"
server_key: "58d0bf52efb466277ace356a83382088"
server_key_iv: "339ed29e724abcd332e365150447e445"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.402379 29574 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:18.412027 29574 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:18.413514 29574 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:18.413648 29574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:18.417563 29574 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:42213
I20260923 23:12:18.417600 29642 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:42213 every 8 connection(s)
I20260923 23:12:18.417954 29574 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:18.419183 29643 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:12:18.419766 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Bootstrap starting.
I20260923 23:12:18.421602 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29574
I20260923 23:12:18.421856 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:34137
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:42213
--webserver_port=39879
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:18.427314 29643 log.cc:824] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:18.428895 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:18.429171 29643 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Bootstrap complete.
I20260923 23:12:18.431465 29643 raft_consensus.cc:348] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:18.431694 29643 raft_consensus.cc:729] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70459954dad94901ac445c7272a9308d, State: Initialized, Role: FOLLOWER
I20260923 23:12:18.431841 29643 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:18.431914 29643 raft_consensus.cc:388] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:18.431962 29643 raft_consensus.cc:482] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:18.432008 29643 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:12:18.432934 29643 raft_consensus.cc:504] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:18.433023 29643 leader_election.cc:302] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70459954dad94901ac445c7272a9308d; no voters:
I20260923 23:12:18.433175 29643 leader_election.cc:288] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:12:18.433229 29647 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:12:18.433442 29643 sys_catalog.cc:564] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:18.433430 29647 raft_consensus.cc:686] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [term 2 LEADER]: Becoming Leader. State: Replica: 70459954dad94901ac445c7272a9308d, State: Running, Role: LEADER
I20260923 23:12:18.433763 29647 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } }
I20260923 23:12:18.434077 29649 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70459954dad94901ac445c7272a9308d. Latest consensus state: current_term: 2 leader_uuid: "70459954dad94901ac445c7272a9308d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } } }
I20260923 23:12:18.434140 29649 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:18.434553 29648 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "70459954dad94901ac445c7272a9308d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70459954dad94901ac445c7272a9308d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 42213 } } }
I20260923 23:12:18.434605 29648 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:18.434942 29643 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260923 23:12:18.435019 29656 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:18.437351 29656 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:18.437665 29656 catalog_manager.cc:1259] Loaded cluster ID: 7d4543b93eb94aafa638bab56abda86d
I20260923 23:12:18.437768 29656 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:18.439592 29656 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:18.439791 29656 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Loaded TSK: 0
I20260923 23:12:18.440073 29656 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:12:18.531638 29645 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:18.531932 29645 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:18.531975 29645 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:18.533684 29645 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:18.533792 29645 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:18.533891 29645 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:18.536056 29645 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:34137
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=39879
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29645
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:18.536521 29645 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:18.536852 29645 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:18.543682 29681 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.543624 29682 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.544060 29645 server_base.cc:1034] running on GCE node
W20260923 23:12:18.544644 29684 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.544942 29645 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:18.545238 29645 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:18.546424 29645 hybrid_clock.cc:648] HybridClock initialized: now 1790205138546401 us; error 38 us; skew 500 ppm
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:18.549718 29645 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:18.550289 29645 webserver.cc:460] Webserver started at http://127.16.122.129:39879/ using document root <none> and password file <none>
I20260923 23:12:18.550562 29645 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:18.550699 29645 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:18.579372 29645 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:18.580449 29692 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.580669 29645 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:18.580777 29645 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "3a8c386bffe0d2251f34b4426a5377ab"
server_key_iv: "eacb86b43c010c789799113feed8aab9"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.581231 29645 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:18.607401 29645 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:18.609776 29645 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:18.610009 29645 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:18.610333 29645 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:18.610743 29645 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:18.610824 29645 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.610922 29645 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:18.610991 29645 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.616770 29645 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:34137
I20260923 23:12:18.617424 29645 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:18.621906 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29645
I20260923 23:12:18.622184 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:33369
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:42213
--webserver_port=41905
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:18.624557 29805 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:34137 every 8 connection(s)
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:18.637995 29809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.625250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.129:60843)
0923 23:12:18.625552 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:18.625598 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:18.626135 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:18.626922 (+ 787us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:18.626926 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:18.627183 (+ 257us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:18.627469 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.627476 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.629167 (+ 1691us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.629169 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.630404 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.630407 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.631356 (+ 949us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.631357 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.631489 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.632103 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:18.632113 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:18.633759 (+ 1646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:18.636159 (+ 2400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.636166 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.636168 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.636443 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.636704 (+ 261us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.636704 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.636705 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.636863 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.637318 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:18.637333 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:18.637493 (+ 160us) client_negotiation.cc:769] Sending connection context
0923 23:12:18.637606 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 23:12:18.637759 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":254,"thread_start_us":83,"threads_started":1}
I20260923 23:12:18.638016 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.625351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:60843 (local address 127.16.122.190:42213)
0923 23:12:18.626305 (+ 954us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:18.626315 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:18.626363 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:12:18.626523 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:18.626532 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:18.626593 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:18.626743 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:18.627609 (+ 866us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.629047 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.630500 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.631140 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.631242 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.633926 (+ 2684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:18.633931 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:18.633931 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:18.633943 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:18.636062 (+ 2119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.636531 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.636534 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.636535 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.636638 (+ 103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.636936 (+ 298us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.636937 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.636937 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.637230 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:18.637311 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:18.637682 (+ 371us) server_negotiation.cc:299] Negotiation successful
0923 23:12:18.637805 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":888,"thread_start_us":81,"threads_started":1}
I20260923 23:12:18.639178 29806 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:18.639276 29806 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.639457 29806 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:18.640331 29806 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:42213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:12:18.640395 29806 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.640448 29806 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:18.640816 29806 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.640882 29806 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:18.641175 29806 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:12:18.731951 29808 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:18.732215 29808 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:18.732249 29808 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:18.734027 29808 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:18.734082 29808 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:18.734119 29808 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:18.736209 29808 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:33369
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=41905
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29808
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:18.736644 29808 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:18.736934 29808 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:18.742619 29818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.747155 29816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.748158 29815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.748538 29808 server_base.cc:1034] running on GCE node
I20260923 23:12:18.748788 29808 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:18.749089 29808 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:18.750363 29808 hybrid_clock.cc:648] HybridClock initialized: now 1790205138750351 us; error 37 us; skew 500 ppm
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:18.756490 29808 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:18.757021 29808 webserver.cc:460] Webserver started at http://127.16.122.130:41905/ using document root <none> and password file <none>
I20260923 23:12:18.757298 29808 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:18.757431 29808 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:18.787200 29808 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.001s sys 0.000s
I20260923 23:12:18.788211 29826 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.788441 29808 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:18.788535 29808 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "6e0f808d43184b7fbda3613b60c8afa7"
format_stamp: "Formatted at 2026-09-23 23:12:17 on dist-test-slave-95hh"
server_key: "69353996a4c23024988fee47817f922c"
server_key_iv: "f9113debe8f21f4015ab0c664444fa94"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.789045 29808 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:18.799057 29808 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:18.801307 29808 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:18.801494 29808 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:18.801776 29808 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:18.802169 29808 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:18.802210 29808 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.802238 29808 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:18.802255 29808 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.808753 29808 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:33369
I20260923 23:12:18.809252 29808 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:12:18.810415 29939 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:33369 every 8 connection(s)
I20260923 23:12:18.814033 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29808
I20260923 23:12:18.814330 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.122.148:37749/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:39971
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:42213
--webserver_port=40217
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:18.818017 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.809964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:36001 (local address 127.16.122.190:42213)
0923 23:12:18.810015 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:18.810017 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:18.810344 (+ 327us) server_negotiation.cc:407] Connection header received
0923 23:12:18.810989 (+ 645us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:18.810990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:18.811003 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:18.811074 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:18.811619 (+ 545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.812104 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.813053 (+ 949us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.813242 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.813296 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.815323 (+ 2027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:18.815324 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:18.815325 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:18.815340 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:18.816574 (+ 1234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.817021 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.817022 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.817023 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.817081 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:18.817330 (+ 249us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:18.817331 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:18.817331 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:18.817460 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:18.817500 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:18.817844 (+ 344us) server_negotiation.cc:299] Negotiation successful
0923 23:12:18.817925 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:12:18.818535 29942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.809982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.130:36001)
0923 23:12:18.810161 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:18.810221 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:18.810862 (+ 641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:18.811171 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:18.811174 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:18.811342 (+ 168us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:18.811526 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.811530 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.812186 (+ 656us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.812186 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.812969 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.812971 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.813333 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.813335 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:18.813441 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:18.813895 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:18.813905 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:18.815235 (+ 1330us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:18.816652 (+ 1417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.816658 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.816660 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.816931 (+ 271us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.817146 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:18.817147 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:18.817147 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:18.817261 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:18.817544 (+ 283us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:18.817550 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:18.817695 (+ 145us) client_negotiation.cc:769] Sending connection context
0923 23:12:18.818186 (+ 491us) client_negotiation.cc:240] Negotiation successful
0923 23:12:18.818330 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":98,"threads_started":1}
I20260923 23:12:18.819185 29940 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:18.819274 29940 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.819444 29940 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:18.820034 29940 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:42213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:12:18.820091 29940 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.820151 29940 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:18.820472 29940 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:18.820541 29940 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:18.820816 29940 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:12:18.914616 29943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:18.914984 29943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:18.915100 29943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:18.916824 29943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:18.916944 29943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:18.917032 29943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:18.919227 29943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.122.148:37749/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:39971
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=40217
--tserver_master_addrs=127.16.122.190:42213
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.29943
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:18.919708 29943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:18.920042 29943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:18.926190 29952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.926319 29949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:18.926354 29950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:18.927162 29943 server_base.cc:1034] running on GCE node
I20260923 23:12:18.927376 29943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:18.927608 29943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:18.928750 29943 hybrid_clock.cc:648] HybridClock initialized: now 1790205138928732 us; error 33 us; skew 500 ppm
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:18.931136 29943 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:18.931602 29943 webserver.cc:460] Webserver started at http://127.16.122.131:40217/ using document root <none> and password file <none>
I20260923 23:12:18.931818 29943 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:18.931877 29943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Sep 23 23:12:18 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:18.944746 29943 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:18.945899 29960 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.946121 29943 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:12:18.946316 29943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "2f3e6565984245c880724f58a341abdf"
format_stamp: "Formatted at 2026-09-23 23:12:18 on dist-test-slave-95hh"
server_key: "eff771637ce9ade1a3ff797e69a2b46b"
server_key_iv: "944d89ed379879d0cf0857afd74d1087"
server_key_version: "kuduclusterkey@0"
I20260923 23:12:18.946887 29943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:18.985611 29943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:18.987748 29943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:18.987913 29943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:18.988154 29943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:18.988543 29943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:18.988601 29943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.988631 29943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:18.988655 29943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:18.995955 29943 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39971
I20260923 23:12:18.996019 30073 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39971 every 8 connection(s)
I20260923 23:12:18.996451 29943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:19.002548 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 29943
Sep 23 23:12:19 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205138, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:19.012002 30076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.997201 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:42213 (local address 127.16.122.131:36807)
0923 23:12:18.997366 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:18.997411 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:18.997872 (+ 461us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:18.998138 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:18.998142 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:18.998311 (+ 169us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:18.998515 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:18.998519 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.999244 (+ 725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:18.999245 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:19.000515 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:19.000518 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:19.001080 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:19.001081 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:19.001229 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:19.001811 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:19.001821 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:19.008127 (+ 6306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:19.010132 (+ 2005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:19.010143 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:19.010146 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:19.010482 (+ 336us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:19.010709 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:19.010711 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:19.010711 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:19.010888 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:19.011237 (+ 349us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:19.011246 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:19.011430 (+ 184us) client_negotiation.cc:769] Sending connection context
0923 23:12:19.011545 (+ 115us) client_negotiation.cc:240] Negotiation successful
0923 23:12:19.011713 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":79,"threads_started":1}
I20260923 23:12:19.012447 29810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:18.997276 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36807 (local address 127.16.122.190:42213)
0923 23:12:18.997344 (+ 68us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:18.997347 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:18.997500 (+ 153us) server_negotiation.cc:407] Connection header received
0923 23:12:18.997970 (+ 470us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:18.997972 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:18.997983 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:18.998062 (+ 79us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:18.998610 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:18.999159 (+ 549us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:19.000619 (+ 1460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:19.000925 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:19.000967 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:19.008230 (+ 7263us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:19.008231 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:19.008232 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:19.008240 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:19.010028 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:19.010572 (+ 544us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:19.010573 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:19.010574 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:19.010637 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:19.010967 (+ 330us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:19.010969 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:19.010969 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:19.011147 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:19.011209 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:19.012296 (+ 1087us) server_negotiation.cc:299] Negotiation successful
0923 23:12:19.012384 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:12:19.012794 30074 heartbeater.cc:347] Connected to a master server at 127.16.122.190:42213
I20260923 23:12:19.012900 30074 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:19.013088 30074 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:19.013726 30074 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:42213 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260923 23:12:19.013805 30074 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:19.013875 30074 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:19.014215 30074 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:19.014293 30074 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
W20260923 23:12:19.014640 30074 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:12:19.017980 30078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:19.002995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56216 (local address 127.16.122.190:42213)
0923 23:12:19.007151 (+ 4156us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:19.007153 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:19.007163 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:12:19.007187 (+ 24us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:19.007229 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:19.007244 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:19.007334 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:19.011878 (+ 4544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:19.012649 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:19.013754 (+ 1105us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:19.014077 (+ 323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:19.014136 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:19.015568 (+ 1432us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:19.015569 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:19.015569 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:19.015575 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:19.016902 (+ 1327us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:19.017313 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:19.017314 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:19.017314 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:19.017369 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:19.017561 (+ 192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:19.017562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:19.017562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:19.017653 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:19.017773 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:19.017852 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 23:12:19.017914 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4130,"thread_start_us":108,"threads_started":1}
Sep 23 23:12:19 dist-test-slave-95hh krb5kdc[28611](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790205139, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:19.641357 29806 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:19.641520 29806 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:19.641997 29607 ts_manager.cc:194] Registered new tserver with Master: b2afc23ab05545a1a23f6cfcdd6a6efb (127.16.122.129:34137)
I20260923 23:12:19.642726 29607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:60843
I20260923 23:12:19.820969 29940 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:19.821131 29940 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:19.821455 29607 ts_manager.cc:194] Registered new tserver with Master: 6e0f808d43184b7fbda3613b60c8afa7 (127.16.122.130:33369)
I20260923 23:12:19.821837 29607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:36001
Sep 23 23:12:19 dist-test-slave-95hh krb5kdc[28611](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205139, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for HTTP/127.16.122.148@KRBTEST.COM
I20260923 23:12:20.014828 30074 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:20.014986 30074 heartbeater.cc:511] Master 127.16.122.190:42213 requested a full tablet report, sending...
I20260923 23:12:20.015393 29607 ts_manager.cc:194] Registered new tserver with Master: 2f3e6565984245c880724f58a341abdf (127.16.122.131:39971)
I20260923 23:12:20.015887 29607 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:36807
I20260923 23:12:20.021776 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:12:20.030768 30096 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.022831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33012 (local address 127.16.122.190:42213)
0923 23:12:20.023003 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.023005 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.023194 (+ 189us) server_negotiation.cc:407] Connection header received
0923 23:12:20.023230 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.023231 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.023243 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.023295 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:20.023948 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.024664 (+ 716us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.026046 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.026335 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.026388 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.027829 (+ 1441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:20.027830 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:20.027831 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:20.027838 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:20.029428 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:20.029782 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:20.029783 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:20.029783 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:20.029838 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:20.029997 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:20.029997 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:20.029998 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:20.030113 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:20.030192 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.030594 (+ 402us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.030672 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":66,"threads_started":1}
I20260923 23:12:20.033196 29607 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:12:20.035671 29607 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:12:20.044257 30105 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:33369 (local address 127.0.0.1:40344)
0923 23:12:20.039866 (+ 1097us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.039874 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:20.039902 (+ 28us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:20.040180 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:20.040180 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:20.040183 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:20.040241 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.040242 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.041596 (+ 1354us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.041597 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.042966 (+ 1369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.042970 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.043967 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.043968 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.044058 (+ 90us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.044074 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 23:12:20.044120 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:12:20.044172 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"thread_start_us":63,"threads_started":1}
I20260923 23:12:20.044747 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40344 (local address 127.16.122.130:33369)
0923 23:12:20.040022 (+ 1164us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.040027 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.040036 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:20.040068 (+ 32us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.040070 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.040085 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.040310 (+ 225us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:20.040397 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.041498 (+ 1101us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.043090 (+ 1592us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.043864 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.044479 (+ 615us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.044549 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.044576 (+ 27us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.044618 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1129,"thread_start_us":43,"threads_started":1}
I20260923 23:12:20.045547 29857 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 (DEFAULT_TABLE table=test-table [id=bb8dd4fe4aa44062aeb90f54c37a1947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:20.045912 29857 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3aaf457496464f8d307b794dcd9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:20.046350 30103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038542 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34137 (local address 127.0.0.1:54052)
0923 23:12:20.039287 (+ 745us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.039328 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:20.039394 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:20.039645 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:20.039648 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:20.039650 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:20.039719 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.039721 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.042499 (+ 2778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.042500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.045210 (+ 2710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.045213 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.046134 (+ 921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.046135 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.046194 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.046198 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:20.046241 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:12:20.046279 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":716,"thread_start_us":84,"threads_started":1}
I20260923 23:12:20.046574 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54052 (local address 127.16.122.129:34137)
0923 23:12:20.039218 (+ 578us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.039223 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.039488 (+ 265us) server_negotiation.cc:407] Connection header received
0923 23:12:20.039530 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.039533 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.039549 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.039627 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:20.039803 (+ 176us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.040821 (+ 1018us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.045302 (+ 4481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.045985 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.046052 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.046101 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.046455 (+ 354us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.046501 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":46,"threads_started":1}
I20260923 23:12:20.047324 29740 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 (DEFAULT_TABLE table=test-table [id=bb8dd4fe4aa44062aeb90f54c37a1947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:20.047678 29740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3aaf457496464f8d307b794dcd9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:20.048090 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39971 (local address 127.0.0.1:49946)
0923 23:12:20.039741 (+ 1196us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.039751 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:20.039802 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:20.041334 (+ 1532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:20.041335 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:20.041336 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:20.041378 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.041380 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.043608 (+ 2228us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.043609 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.046962 (+ 3353us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.046965 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.047893 (+ 928us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.047894 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.047954 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.047958 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:20.047984 (+ 26us) client_negotiation.cc:240] Negotiation successful
0923 23:12:20.048021 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1177,"thread_start_us":95,"threads_started":1}
I20260923 23:12:20.048462 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.038958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49946 (local address 127.16.122.131:39971)
0923 23:12:20.039111 (+ 153us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.039116 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.040904 (+ 1788us) server_negotiation.cc:407] Connection header received
0923 23:12:20.040940 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.040943 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.040958 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.041020 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:20.041481 (+ 461us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.042420 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.047072 (+ 4652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.047802 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.048233 (+ 431us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.048306 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.048332 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.048374 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":95,"threads_started":1}
I20260923 23:12:20.049094 30008 tablet_service.cc:1467] Processing CreateTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 (DEFAULT_TABLE table=test-table [id=bb8dd4fe4aa44062aeb90f54c37a1947]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:20.049417 30008 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9f3aaf457496464f8d307b794dcd9cc9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:20.050731 30109 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Bootstrap starting.
I20260923 23:12:20.051302 30108 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Bootstrap starting.
I20260923 23:12:20.051381 30109 tablet_bootstrap.cc:654] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:20.051700 30109 log.cc:824] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:20.052035 30108 tablet_bootstrap.cc:654] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:20.052315 30108 log.cc:824] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:20.052690 30111 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Bootstrap starting.
I20260923 23:12:20.053164 30111 tablet_bootstrap.cc:654] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:20.053399 30111 log.cc:824] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:20.054100 30111 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: No bootstrap required, opened a new log
I20260923 23:12:20.054095 30108 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: No bootstrap required, opened a new log
I20260923 23:12:20.054159 30108 ts_tablet_manager.cc:1397] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:12:20.054201 30111 ts_tablet_manager.cc:1397] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:20.056664 30111 raft_consensus.cc:348] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.056732 30108 raft_consensus.cc:348] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.056851 30108 raft_consensus.cc:374] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:20.056905 30108 raft_consensus.cc:729] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6e0f808d43184b7fbda3613b60c8afa7, State: Initialized, Role: FOLLOWER
I20260923 23:12:20.056977 30111 raft_consensus.cc:374] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:20.057073 30111 raft_consensus.cc:729] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f3e6565984245c880724f58a341abdf, State: Initialized, Role: FOLLOWER
I20260923 23:12:20.057072 30108 consensus_queue.cc:260] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.057374 30108 ts_tablet_manager.cc:1428] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:12:20.057518 29940 heartbeater.cc:503] Master 127.16.122.190:42213 was elected leader, sending a full tablet report...
I20260923 23:12:20.057488 30111 consensus_queue.cc:260] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.057726 30111 ts_tablet_manager.cc:1428] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:12:20.058265 30109 tablet_bootstrap.cc:492] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: No bootstrap required, opened a new log
I20260923 23:12:20.058444 30109 ts_tablet_manager.cc:1397] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.000s
W20260923 23:12:20.060345 29941 tablet.cc:2369] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:12:20.060274 30109 raft_consensus.cc:348] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.060423 30109 raft_consensus.cc:374] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:20.060452 30109 raft_consensus.cc:729] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2afc23ab05545a1a23f6cfcdd6a6efb, State: Initialized, Role: FOLLOWER
I20260923 23:12:20.060575 30109 consensus_queue.cc:260] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.060825 29806 heartbeater.cc:503] Master 127.16.122.190:42213 was elected leader, sending a full tablet report...
I20260923 23:12:20.060891 30074 heartbeater.cc:503] Master 127.16.122.190:42213 was elected leader, sending a full tablet report...
I20260923 23:12:20.060823 30109 ts_tablet_manager.cc:1428] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:12:20.109660 30115 raft_consensus.cc:482] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:20.109822 30115 raft_consensus.cc:504] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.110553 30115 leader_election.cc:288] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6e0f808d43184b7fbda3613b60c8afa7 (127.16.122.130:33369), b2afc23ab05545a1a23f6cfcdd6a6efb (127.16.122.129:34137)
I20260923 23:12:20.114217 30104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.110723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:41257 (local address 127.16.122.129:34137)
0923 23:12:20.110782 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.110784 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.110795 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:12:20.110890 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.110892 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.110902 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.110931 (+ 29us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:20.111177 (+ 246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.111880 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.113297 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.113825 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.113874 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.113883 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.114098 (+ 215us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.114145 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20260923 23:12:20.114713 30118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.110419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34137 (local address 127.16.122.131:41257)
0923 23:12:20.110717 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.110729 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:20.110789 (+ 60us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:20.110997 (+ 208us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:20.110997 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:20.110999 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:20.111100 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.111103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.111980 (+ 877us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.111980 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.113183 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.113186 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.113957 (+ 771us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.113958 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.114023 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.114027 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:20.114553 (+ 526us) client_negotiation.cc:240] Negotiation successful
0923 23:12:20.114609 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":273,"thread_start_us":94,"threads_started":1}
I20260923 23:12:20.114850 30106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.110853 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:57845 (local address 127.16.122.130:33369)
0923 23:12:20.110895 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.110897 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.111077 (+ 180us) server_negotiation.cc:407] Connection header received
0923 23:12:20.111181 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.111183 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.111192 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.111219 (+ 27us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:20.111435 (+ 216us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.112140 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.113899 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.114441 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.114490 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.114497 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.114720 (+ 223us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.114776 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:12:20.115180 30117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.110427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:33369 (local address 127.16.122.131:57845)
0923 23:12:20.110981 (+ 554us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.110991 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:20.111105 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:20.111284 (+ 179us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:20.111284 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:20.111286 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:20.111356 (+ 70us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.111358 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.112236 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.112237 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.113794 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:20.113797 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.114557 (+ 760us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.114558 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:20.114636 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.114640 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:20.115058 (+ 418us) client_negotiation.cc:240] Negotiation successful
0923 23:12:20.115102 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":536,"thread_start_us":122,"threads_started":1}
I20260923 23:12:20.115224 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3aaf457496464f8d307b794dcd9cc9" candidate_uuid: "2f3e6565984245c880724f58a341abdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" is_pre_election: true
I20260923 23:12:20.115420 29760 raft_consensus.cc:2454] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3e6565984245c880724f58a341abdf in term 0.
I20260923 23:12:20.115557 29894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3aaf457496464f8d307b794dcd9cc9" candidate_uuid: "2f3e6565984245c880724f58a341abdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0f808d43184b7fbda3613b60c8afa7" is_pre_election: true
I20260923 23:12:20.115693 29894 raft_consensus.cc:2454] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2f3e6565984245c880724f58a341abdf in term 0.
I20260923 23:12:20.115883 29961 leader_election.cc:302] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3e6565984245c880724f58a341abdf, b2afc23ab05545a1a23f6cfcdd6a6efb; no voters:
I20260923 23:12:20.116024 30115 raft_consensus.cc:2781] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:12:20.116091 30115 raft_consensus.cc:482] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:20.116117 30115 raft_consensus.cc:3037] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:20.117043 30115 raft_consensus.cc:504] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.117192 30115 leader_election.cc:288] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [CANDIDATE]: Term 1 election: Requested vote from peers 6e0f808d43184b7fbda3613b60c8afa7 (127.16.122.130:33369), b2afc23ab05545a1a23f6cfcdd6a6efb (127.16.122.129:34137)
I20260923 23:12:20.117341 29894 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3aaf457496464f8d307b794dcd9cc9" candidate_uuid: "2f3e6565984245c880724f58a341abdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6e0f808d43184b7fbda3613b60c8afa7"
I20260923 23:12:20.117420 29894 raft_consensus.cc:3037] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:20.117516 29760 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9f3aaf457496464f8d307b794dcd9cc9" candidate_uuid: "2f3e6565984245c880724f58a341abdf" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb"
I20260923 23:12:20.117599 29760 raft_consensus.cc:3037] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:20.118254 29894 raft_consensus.cc:2454] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3e6565984245c880724f58a341abdf in term 1.
I20260923 23:12:20.118466 29760 raft_consensus.cc:2454] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f3e6565984245c880724f58a341abdf in term 1.
I20260923 23:12:20.118467 29963 leader_election.cc:302] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f3e6565984245c880724f58a341abdf, 6e0f808d43184b7fbda3613b60c8afa7; no voters:
I20260923 23:12:20.118575 30115 raft_consensus.cc:2781] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:20.118711 30115 raft_consensus.cc:686] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 1 LEADER]: Becoming Leader. State: Replica: 2f3e6565984245c880724f58a341abdf, State: Running, Role: LEADER
I20260923 23:12:20.118801 30115 consensus_queue.cc:237] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } }
I20260923 23:12:20.119620 29607 catalog_manager.cc:5495] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf reported cstate change: term changed from 0 to 1, leader changed from <none> to 2f3e6565984245c880724f58a341abdf (127.16.122.131). New cstate: current_term: 1 leader_uuid: "2f3e6565984245c880724f58a341abdf" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f3e6565984245c880724f58a341abdf" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39971 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 } health_report { overall_health: UNKNOWN } } }
W20260923 23:12:20.126472 29807 tablet.cc:2369] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:12:20.137501 30107 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.134270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49950 (local address 127.16.122.131:39971)
0923 23:12:20.134321 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:20.134323 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:20.134329 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:12:20.134368 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:20.134369 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:20.134389 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:20.134446 (+ 57us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:12:20.134616 (+ 170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.135356 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.136426 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:20.136716 (+ 290us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:20.136765 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:20.136990 (+ 225us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:12:20.137118 (+ 128us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:12:20.137182 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:20.137407 (+ 225us) server_negotiation.cc:299] Negotiation successful
0923 23:12:20.137427 (+ 20us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:12:20.139410 29760 raft_consensus.cc:1261] T 9f3aaf457496464f8d307b794dcd9cc9 P b2afc23ab05545a1a23f6cfcdd6a6efb [term 1 FOLLOWER]: Refusing update from remote peer 2f3e6565984245c880724f58a341abdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:20.139533 29894 raft_consensus.cc:1261] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 1 FOLLOWER]: Refusing update from remote peer 2f3e6565984245c880724f58a341abdf: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:20.139719 30115 consensus_queue.cc:1035] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "b2afc23ab05545a1a23f6cfcdd6a6efb" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 34137 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:20.139847 30119 consensus_queue.cc:1035] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [LEADER]: Connected to new peer: Peer: permanent_uuid: "6e0f808d43184b7fbda3613b60c8afa7" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 33369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:20.142666 30122 mvcc.cc:205] Tried to move back new op lower bound from 7332680254008168448 to 7332680253927219200. Current Snapshot: MvccSnapshot[applied={T|T < 7332680254008168448}]
I20260923 23:12:20.143283 30123 mvcc.cc:205] Tried to move back new op lower bound from 7332680254008168448 to 7332680253927219200. Current Snapshot: MvccSnapshot[applied={T|T < 7332680254008168448}]
I20260923 23:12:20.144979 30124 mvcc.cc:205] Tried to move back new op lower bound from 7332680254008168448 to 7332680253927219200. Current Snapshot: MvccSnapshot[applied={T|T < 7332680254008168448}]
I20260923 23:12:20.145880 29606 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:20.145972 29606 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33012:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:20.146716 29606 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 70459954dad94901ac445c7272a9308d: Sending DeleteTablet for 3 replicas of tablet 9f3aaf457496464f8d307b794dcd9cc9
I20260923 23:12:20.147158 29857 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:20 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40344
I20260923 23:12:20.147374 29740 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:20 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:54052
I20260923 23:12:20.147838 30008 tablet_service.cc:1514] Processing DeleteTablet for tablet 9f3aaf457496464f8d307b794dcd9cc9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:20 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49946
I20260923 23:12:20.148237 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29645
I20260923 23:12:20.152719 30133 tablet_replica.cc:330] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: stopping tablet replica
I20260923 23:12:20.152868 30131 tablet_replica.cc:330] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: stopping tablet replica
I20260923 23:12:20.153048 30131 raft_consensus.cc:2229] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:20.153172 30133 raft_consensus.cc:2229] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:12:20.153358 30133 raft_consensus.cc:2258] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:20.153544 30131 raft_consensus.cc:2258] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:20.154227 30133 ts_tablet_manager.cc:1905] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:20.154516 30131 ts_tablet_manager.cc:1905] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:20.160465 30133 ts_tablet_manager.cc:1918] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:20.160693 30131 ts_tablet_manager.cc:1918] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:20.160820 30133 log.cc:1196] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/9f3aaf457496464f8d307b794dcd9cc9
I20260923 23:12:20.160936 30131 log.cc:1196] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/9f3aaf457496464f8d307b794dcd9cc9
I20260923 23:12:20.161309 30133 ts_tablet_manager.cc:1939] T 9f3aaf457496464f8d307b794dcd9cc9 P 2f3e6565984245c880724f58a341abdf: Deleting consensus metadata
I20260923 23:12:20.161602 30131 ts_tablet_manager.cc:1939] T 9f3aaf457496464f8d307b794dcd9cc9 P 6e0f808d43184b7fbda3613b60c8afa7: Deleting consensus metadata
I20260923 23:12:20.162106 29593 catalog_manager.cc:4841] TS 6e0f808d43184b7fbda3613b60c8afa7 (127.16.122.130:33369): tablet 9f3aaf457496464f8d307b794dcd9cc9 (table test-table [id=bb8dd4fe4aa44062aeb90f54c37a1947]) successfully deleted
I20260923 23:12:20.162642 29593 catalog_manager.cc:4841] TS 2f3e6565984245c880724f58a341abdf (127.16.122.131:39971): tablet 9f3aaf457496464f8d307b794dcd9cc9 (table test-table [id=bb8dd4fe4aa44062aeb90f54c37a1947]) successfully deleted
W20260923 23:12:20.162997 29591 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:34137 (error 108)
I20260923 23:12:20.163857 30102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:20.163727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:34137 (local address 127.0.0.1:54058)
0923 23:12:20.163770 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:20.163821 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:34137: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 23:12:20.163858 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29808
W20260923 23:12:20.165522 29591 catalog_manager.cc:4584] TS b2afc23ab05545a1a23f6cfcdd6a6efb (127.16.122.129:34137): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 9f3aaf457496464f8d307b794dcd9cc9: Network error: Client connection negotiation failed: client connection to 127.16.122.129:34137: connect: Connection refused (error 111)
I20260923 23:12:20.168972 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29943
I20260923 23:12:20.174388 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 29574
I20260923 23:12:20.179234 16874 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260923 23:12:20.513679 16874 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260923 23:12:20.513782 16874 mini_ranger.cc:66] Stopping Ranger...
I20260923 23:12:20.847962 16874 mini_ranger.cc:68] Stopped Ranger
2026-09-23 23:12:20.848 UTC [28641] LOG: received smart shutdown request
2026-09-23 23:12:20.852 UTC [28641] LOG: background worker "logical replication launcher" (PID 28650) exited with exit code 1
2026-09-23 23:12:20.852 UTC [28645] LOG: shutting down
2026-09-23 23:12:20.998 UTC [28641] LOG: database system is shut down
2026-09-23T23:12:20Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (45736 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:12:21 dist-test-slave-95hh krb5kdc[30143](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:12:21 dist-test-slave-95hh krb5kdc[30143](info): set up 2 sockets
Sep 23 23:12:21 dist-test-slave-95hh krb5kdc[30143](info): commencing operation
krb5kdc: starting...
W20260923 23:12:22.030669 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:12:22Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:12:22Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:12:22.139923 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:39643
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:22.218443 30156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:22.218647 30156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:22.218675 30156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:22.219810 30156 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:22.219847 30156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:22.219864 30156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:22.219880 30156 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:22.219895 30156 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:22.221388 30156 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:39643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30156
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:22.221624 30156 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:22.221832 30156 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:22.226449 30164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:22.226540 30156 server_base.cc:1034] running on GCE node
W20260923 23:12:22.226421 30162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:22.226433 30161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:22.226881 30156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:22.227164 30156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:22.228341 30156 hybrid_clock.cc:648] HybridClock initialized: now 1790205142228322 us; error 34 us; skew 500 ppm
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:22.234125 30156 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:22.234596 30156 webserver.cc:460] Webserver started at http://127.16.122.190:39363/ using document root <none> and password file <none>
I20260923 23:12:22.234800 30156 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:22.234848 30156 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:22.234978 30156 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:22.236063 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "8a84dd48ecb336ea167ae864828f2c7f"
server_key_iv: "69627ace5a12be12f20c34eda3ac7789"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.236377 30156 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "8a84dd48ecb336ea167ae864828f2c7f"
server_key_iv: "69627ace5a12be12f20c34eda3ac7789"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.237589 30156 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:22.238308 30171 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:22.238466 30156 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:22.238546 30156 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "8a84dd48ecb336ea167ae864828f2c7f"
server_key_iv: "69627ace5a12be12f20c34eda3ac7789"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.238605 30156 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:22.258275 30156 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:22.260452 30156 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:22.260587 30156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:22.264230 30156 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:39643
I20260923 23:12:22.264266 30223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:39643 every 8 connection(s)
I20260923 23:12:22.264595 30156 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:22.265375 30224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:22.266871 30224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:22.274632 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30156
I20260923 23:12:22.274727 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:12:22.275022 16874 external_mini_cluster.cc:1428] Setting key a0aef762c6991cc03c50c24ea8a50655
I20260923 23:12:22.275511 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.267490 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39643 (local address 127.0.0.1:44816)
0923 23:12:22.267642 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.267680 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.268118 (+ 438us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.268359 (+ 241us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.268361 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.268594 (+ 233us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.268748 (+ 154us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.268752 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.269686 (+ 934us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.269687 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.270419 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.270421 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.270894 (+ 473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.270896 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.270964 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.271427 (+ 463us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.271435 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.272730 (+ 1295us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.274116 (+ 1386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.274122 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.274124 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.274295 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.274521 (+ 226us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.274522 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.274523 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.274595 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.274940 (+ 345us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.274945 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.275007 (+ 62us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.275116 (+ 109us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.275336 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112,"mutex_wait_us":84,"thread_start_us":67,"threads_started":1}
I20260923 23:12:22.275513 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.267517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44816 (local address 127.16.122.190:39643)
0923 23:12:22.267679 (+ 162us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.267689 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.267766 (+ 77us) server_negotiation.cc:407] Connection header received
0923 23:12:22.268241 (+ 475us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.268243 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.268260 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.268331 (+ 71us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.268817 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.269618 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.270479 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.270800 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.270877 (+ 77us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.272804 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.272811 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.272811 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.272817 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.274046 (+ 1229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.274363 (+ 317us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.274365 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.274366 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.274451 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.274646 (+ 195us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.274647 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.274647 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.274881 (+ 234us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.274953 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.275139 (+ 186us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.275262 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":81,"threads_started":1}
I20260923 23:12:22.276209 30224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:12:22.276568 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.276451 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38462)
0923 23:12:22.276498 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.276541 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:12:22.276695 30173 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
I20260923 23:12:22.277069 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.276985 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38472)
0923 23:12:22.277025 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.277050 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:12:22.277158 30224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36635: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.16.122.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
I20260923 23:12:22.295643 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:36635
--webserver_interface=127.16.122.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:22.326373 30224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } attempt: 1
I20260923 23:12:22.326874 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.326741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38486)
0923 23:12:22.326802 (+ 61us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.326845 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
I20260923 23:12:22.327237 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.327154 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38496)
0923 23:12:22.327195 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.327218 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 23:12:22.327320 30224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36635: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
W20260923 23:12:22.373754 30231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:22.373970 30231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:22.373991 30231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:22.375136 30231 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:22.375172 30231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:22.375185 30231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:22.375197 30231 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:22.375208 30231 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:22.376662 30231 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:36635
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.122.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30231
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:22.376881 30231 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:22.377095 30231 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:22.381619 30236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:22.381619 30239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:22.381639 30237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:22.381681 30231 server_base.cc:1034] running on GCE node
I20260923 23:12:22.382066 30231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:22.382339 30231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:22.383469 30231 hybrid_clock.cc:648] HybridClock initialized: now 1790205142383455 us; error 28 us; skew 500 ppm
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:22.385744 30231 init.cc:377] Logged in from keytab as kudu/127.16.122.189@KRBTEST.COM (short username kudu)
I20260923 23:12:22.386157 30231 webserver.cc:460] Webserver started at http://127.16.122.189:33231/ using document root <none> and password file <none>
I20260923 23:12:22.386336 30231 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:22.386373 30231 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:22.386466 30231 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:22.387485 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data/instance:
uuid: "567ecfbb4b1f4b08bdc86c3db2070667"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "16ccbc45fa4201606f6cc782fd4347f7"
server_key_iv: "221ed34d6f1a42d0a0709130260eb50f"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.387765 30231 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal/instance:
uuid: "567ecfbb4b1f4b08bdc86c3db2070667"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "16ccbc45fa4201606f6cc782fd4347f7"
server_key_iv: "221ed34d6f1a42d0a0709130260eb50f"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.388885 30231 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:22.389518 30246 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:22.389662 30231 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:22.389736 30231 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "567ecfbb4b1f4b08bdc86c3db2070667"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "16ccbc45fa4201606f6cc782fd4347f7"
server_key_iv: "221ed34d6f1a42d0a0709130260eb50f"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.389802 30231 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:22.395442 30224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } attempt: 2
I20260923 23:12:22.395812 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.395683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38498)
0923 23:12:22.395737 (+ 54us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.395777 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 23:12:22.396111 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.396038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38500)
0923 23:12:22.396075 (+ 37us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.396096 (+ 21us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
W20260923 23:12:22.396193 30224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36635: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36635: connect: Connection refused (error 111)
I20260923 23:12:22.406497 30231 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:22.408401 30231 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:22.408550 30231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:22.412205 30231 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.189:36635
I20260923 23:12:22.412243 30298 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.189:36635 every 8 connection(s)
I20260923 23:12:22.412555 30231 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
I20260923 23:12:22.413355 30299 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:22.414829 30299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:22.420223 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30231
I20260923 23:12:22.420305 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-1/wal/instance
I20260923 23:12:22.420467 16874 external_mini_cluster.cc:1428] Setting key 3ce6966fd0682b4a4546eda8d7696ddd
I20260923 23:12:22.422755 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.415530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44826 (local address 127.16.122.190:39643)
0923 23:12:22.415590 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.415592 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.415781 (+ 189us) server_negotiation.cc:407] Connection header received
0923 23:12:22.416231 (+ 450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.416231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.416240 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.416295 (+ 55us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.416897 (+ 602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.417382 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.418278 (+ 896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.418468 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.418511 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.420395 (+ 1884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.420397 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.420397 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.420405 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.421567 (+ 1162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.421909 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.421910 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.421910 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.421964 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.422179 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.422179 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.422180 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.422288 (+ 108us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.422340 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.422585 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.422666 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:12:22.422915 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.415505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39643 (local address 127.0.0.1:44826)
0923 23:12:22.415665 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.415706 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.416154 (+ 448us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.416351 (+ 197us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.416354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.416636 (+ 282us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.416829 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.416833 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.417444 (+ 611us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.417445 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.418221 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.418223 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.418520 (+ 297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.418520 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.418618 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.419072 (+ 454us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.419081 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.420315 (+ 1234us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.421637 (+ 1322us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.421644 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.421646 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.421846 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.422019 (+ 173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.422019 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.422020 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.422125 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.422340 (+ 215us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.422346 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.422474 (+ 128us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.422558 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.422702 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":76,"threads_started":1}
I20260923 23:12:22.423398 30299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.122.188@KRBTEST.COM; defaulting to no policy
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.188@KRBTEST.COM" created.
I20260923 23:12:22.430275 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.423709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38510 (local address 127.16.122.189:36635)
0923 23:12:22.423848 (+ 139us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.423852 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.423867 (+ 15us) server_negotiation.cc:407] Connection header received
0923 23:12:22.423882 (+ 15us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.423884 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.423894 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.423952 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.424184 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.424864 (+ 680us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.425485 (+ 621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.425840 (+ 355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.425898 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.427504 (+ 1606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.427511 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.427511 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.427517 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.428860 (+ 1343us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.429152 (+ 292us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.429154 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.429155 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.429202 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.429709 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.429710 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.429710 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.429850 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.429913 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.430090 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.430177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112,"thread_start_us":76,"threads_started":1}
I20260923 23:12:22.430336 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.423636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38510)
0923 23:12:22.423702 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.423710 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.423742 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.423965 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.423966 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.424086 (+ 120us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.424126 (+ 40us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.424128 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.424938 (+ 810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.424939 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.425421 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.425423 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.425902 (+ 479us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.425903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.425970 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.426321 (+ 351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.426326 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.427422 (+ 1096us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.428929 (+ 1507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.428930 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.428930 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.429094 (+ 164us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.429250 (+ 156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.429251 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.429251 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.429307 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.429929 (+ 622us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.429930 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.429988 (+ 58us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.430062 (+ 74us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.430214 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51,"mutex_wait_us":49}
I20260923 23:12:22.430958 30299 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:12:22.431293 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.431191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35504)
0923 23:12:22.431230 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.431272 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 23:12:22.431416 30249 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
I20260923 23:12:22.431804 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.431729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35520)
0923 23:12:22.431768 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.431790 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:12:22.431891 30299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:43427: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
I20260923 23:12:22.443066 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:43427
--webserver_interface=127.16.122.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:35191
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:22.481117 30299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } attempt: 1
I20260923 23:12:22.481637 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.481502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35534)
0923 23:12:22.481557 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.481604 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 23:12:22.481995 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.481896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35548)
0923 23:12:22.481940 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.481968 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:12:22.482092 30299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:43427: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
I20260923 23:12:22.487329 30224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } attempt: 3
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:12:22.493744 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.487640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38514 (local address 127.16.122.189:36635)
0923 23:12:22.487695 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.487697 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.487708 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:12:22.487765 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.487766 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.487777 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.487822 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.488118 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.488638 (+ 520us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.489306 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.489508 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.489562 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.491467 (+ 1905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.491469 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.491469 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.491476 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.492752 (+ 1276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.493051 (+ 299us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.493052 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.493053 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.493107 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.493325 (+ 218us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.493325 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.493326 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.493426 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.493471 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.493622 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.493680 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:12:22.493801 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.487591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38514)
0923 23:12:22.487644 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.487654 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.487702 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.487833 (+ 131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.487834 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.487985 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.488049 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.488051 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.488713 (+ 662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.488714 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.489228 (+ 514us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.489230 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.489569 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.489570 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.489639 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.490081 (+ 442us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.490085 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.491393 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.492833 (+ 1440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.492834 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.492834 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.492994 (+ 160us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.493178 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.493179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.493180 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.493258 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.493480 (+ 222us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.493481 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.493534 (+ 53us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.493607 (+ 73us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.493708 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":33}
I20260923 23:12:22.494165 30224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:12:22.494506 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.494394 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35560)
0923 23:12:22.494439 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.494482 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:12:22.494788 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.494716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35564)
0923 23:12:22.494751 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.494773 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:12:22.494867 30224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:43427: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
W20260923 23:12:22.522928 30306 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:22.523175 30306 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:22.523196 30306 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:22.524341 30306 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:22.524384 30306 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:22.524396 30306 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:22.524408 30306 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:22.524430 30306 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:22.525991 30306 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:35191
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:39643,127.16.122.189:36635,127.16.122.188:43427
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:43427
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.122.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30306
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:22.526006 30224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } attempt: 1
I20260923 23:12:22.526234 30306 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:22.526443 30306 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:12:22.526446 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.526321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35572)
0923 23:12:22.526384 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.526418 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:12:22.526762 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.526677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35584)
0923 23:12:22.526719 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.526743 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:12:22.526844 30224 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:43427: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
I20260923 23:12:22.531299 30306 server_base.cc:1034] running on GCE node
W20260923 23:12:22.531271 30311 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:22.531262 30312 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:22.531275 30314 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:22.531798 30306 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:22.532074 30306 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:22.533221 30306 hybrid_clock.cc:648] HybridClock initialized: now 1790205142533210 us; error 28 us; skew 500 ppm
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:22.535598 30306 init.cc:377] Logged in from keytab as kudu/127.16.122.188@KRBTEST.COM (short username kudu)
I20260923 23:12:22.536008 30306 webserver.cc:460] Webserver started at http://127.16.122.188:46143/ using document root <none> and password file <none>
I20260923 23:12:22.536208 30306 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:22.536255 30306 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:22.536377 30306 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:22.537423 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data/instance:
uuid: "0181e827f04d4a7aab0bd4ade60e780b"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "3f5b6604df0a5765f5d97de5f4a11863"
server_key_iv: "6281f6e233f92a8b11338db162375aa7"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.537741 30306 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal/instance:
uuid: "0181e827f04d4a7aab0bd4ade60e780b"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "3f5b6604df0a5765f5d97de5f4a11863"
server_key_iv: "6281f6e233f92a8b11338db162375aa7"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.538935 30306 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:12:22.539597 30321 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:22.539742 30306 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:22.539826 30306 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "0181e827f04d4a7aab0bd4ade60e780b"
format_stamp: "Formatted at 2026-09-23 23:12:22 on dist-test-slave-95hh"
server_key: "3f5b6604df0a5765f5d97de5f4a11863"
server_key_iv: "6281f6e233f92a8b11338db162375aa7"
server_key_version: "encryptionkey@0"
I20260923 23:12:22.539887 30306 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:22.550246 30299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } attempt: 2
I20260923 23:12:22.550653 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.550511 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35592)
0923 23:12:22.550573 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.550614 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:12:22.550689 30306 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:22.550995 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.550905 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35602)
0923 23:12:22.550947 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.550975 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 23:12:22.551096 30299 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:43427: Network error: Client connection negotiation failed: client connection to 127.16.122.188:43427: connect: Connection refused (error 111)
I20260923 23:12:22.552879 30306 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:22.552999 30306 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:22.556752 30306 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.188:43427
I20260923 23:12:22.556792 30373 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.188:43427 every 8 connection(s)
I20260923 23:12:22.557125 30306 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
I20260923 23:12:22.557981 30374 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:22.559719 30374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:22.567334 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30306
I20260923 23:12:22.567440 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACert.1790204999454807-16874-0/minicluster-data/master-2/wal/instance
I20260923 23:12:22.567623 16874 external_mini_cluster.cc:1428] Setting key 15714c2ef5207d4fdff357cfde8b3249
I20260923 23:12:22.568068 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.560449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44830 (local address 127.16.122.190:39643)
0923 23:12:22.560531 (+ 82us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.560533 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.560676 (+ 143us) server_negotiation.cc:407] Connection header received
0923 23:12:22.561172 (+ 496us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.561173 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.561185 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.561238 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.561856 (+ 618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.562324 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.563335 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.563537 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.563592 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.565642 (+ 2050us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.565644 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.565644 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.565650 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.566744 (+ 1094us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.567115 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.567116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.567116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.567166 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.567409 (+ 243us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.567410 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.567410 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.567547 (+ 137us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.567615 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.567888 (+ 273us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.567976 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 23:12:22.568413 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.560364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:39643 (local address 127.0.0.1:44830)
0923 23:12:22.560547 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.560592 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.561073 (+ 481us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.561308 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.561312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.561568 (+ 256us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.561766 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.561770 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.562405 (+ 635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.562406 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.563260 (+ 854us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.563262 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.563604 (+ 342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.563605 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.563725 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.564203 (+ 478us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.564212 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.565562 (+ 1350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.566807 (+ 1245us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.566813 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.566815 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.567028 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.567225 (+ 197us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.567226 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.567226 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.567333 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.567606 (+ 273us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.567611 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.567761 (+ 150us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.567855 (+ 94us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.568183 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":78,"threads_started":1}
I20260923 23:12:22.568948 30374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:12:22.575683 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.569211 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36635 (local address 127.0.0.1:38522)
0923 23:12:22.569252 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.569261 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.569293 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.569441 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.569442 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.569569 (+ 127us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.569614 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.569616 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.570392 (+ 776us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.570393 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.570997 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.570998 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.571397 (+ 399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.571397 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.571443 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.571876 (+ 433us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.571880 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.573136 (+ 1256us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.574630 (+ 1494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.574631 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.574631 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.574846 (+ 215us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.575083 (+ 237us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.575084 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.575084 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.575139 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.575405 (+ 266us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.575406 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.575442 (+ 36us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.575488 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.575589 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:12:22.575706 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.569286 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38522 (local address 127.16.122.189:36635)
0923 23:12:22.569348 (+ 62us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.569350 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.569358 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:12:22.569380 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.569381 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.569390 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.569433 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.569677 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.570321 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.571107 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.571327 (+ 220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.571391 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.573248 (+ 1857us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.573250 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.573250 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.573258 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.574559 (+ 1301us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.574930 (+ 371us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.574931 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.574932 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.574994 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.575193 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.575194 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.575194 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.575335 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.575407 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.575529 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.575606 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:12:22.576259 30374 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:12:22.576979 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.567955 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35606 (local address 127.16.122.188:43427)
0923 23:12:22.568131 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.568136 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.568162 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:12:22.568237 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.568242 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.568255 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.568316 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.569015 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.569803 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.570707 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.571190 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.571238 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.573845 (+ 2607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.573857 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.573857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.573865 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.575593 (+ 1728us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.575991 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.575995 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.575995 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.576047 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.576322 (+ 275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.576323 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.576323 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.576522 (+ 199us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.576668 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.576773 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.576877 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:22.583403 30376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.576519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35614)
0923 23:12:22.576564 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.576573 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.576603 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.576824 (+ 221us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.576825 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.576964 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.577006 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.577007 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.577685 (+ 678us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.577686 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.578256 (+ 570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.578258 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.578646 (+ 388us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.578646 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.578701 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.579128 (+ 427us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.579132 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.580321 (+ 1189us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.582516 (+ 2195us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.582517 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.582517 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.582662 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.582834 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.582835 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.582835 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.582896 (+ 61us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.583170 (+ 274us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.583171 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.583206 (+ 35us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.583262 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.583337 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 23:12:22.583593 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.576554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35614 (local address 127.16.122.188:43427)
0923 23:12:22.576720 (+ 166us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.576722 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.576727 (+ 5us) server_negotiation.cc:407] Connection header received
0923 23:12:22.576740 (+ 13us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.576741 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.576750 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.577071 (+ 321us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.577092 (+ 21us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.577616 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.578335 (+ 719us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.578585 (+ 250us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.578632 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.580385 (+ 1753us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.580386 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.580386 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.580392 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.582442 (+ 2050us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.582725 (+ 283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.582726 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.582727 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.582778 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.582950 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.582950 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.582951 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.583108 (+ 157us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.583179 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.583286 (+ 107us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.583391 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":30,"server-negotiator.queue_time_us":148,"thread_start_us":113,"threads_started":1}
I20260923 23:12:22.584486 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.577831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44832 (local address 127.16.122.190:39643)
0923 23:12:22.577872 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.577874 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.577880 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:12:22.577903 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.577904 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.577912 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.577956 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.578389 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.578945 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.579731 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.579915 (+ 184us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.579965 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.582070 (+ 2105us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.582071 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.582071 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.582080 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.583530 (+ 1450us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.583859 (+ 329us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.583860 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.583860 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.583914 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.584075 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.584075 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.584076 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.584172 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.584300 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.584368 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.584427 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:12:22.585155 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: Bootstrap starting.
I20260923 23:12:22.585871 30374 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:22.586154 30374 log.cc:824] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:22.586948 30374 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: No bootstrap required, opened a new log
I20260923 23:12:22.589088 30374 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.589214 30374 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:22.589237 30374 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0181e827f04d4a7aab0bd4ade60e780b, State: Initialized, Role: FOLLOWER
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:12:22.589352 30374 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.589629 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.589704 30374 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:22.589712 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:12:22.591663 30392 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:12:22.591725 30392 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:12:22.592711 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.585049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.16.122.189:36635)
0923 23:12:22.585084 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.585086 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.585095 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:22.585116 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.585117 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.585126 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.585168 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.585664 (+ 496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.586151 (+ 487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.587183 (+ 1032us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.587374 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.587430 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.589846 (+ 2416us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.589847 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.589848 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.589855 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.591428 (+ 1573us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.591835 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.591836 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.591836 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.591889 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.592130 (+ 241us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.592131 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.592131 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.592326 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.592497 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.592539 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.592618 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:12:22.594110 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:12:22.592312 30306 master_runner.cc:186] Error getting master registration for 127.16.122.190:39643: OK, instance_id { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" instance_seqno: 1790205142263199 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:12:22.594480 30306 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:12:22.601981 30224 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } attempt: 2
I20260923 23:12:22.602744 30227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.595484 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44842 (local address 127.16.122.190:39643)
0923 23:12:22.595521 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.595523 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.595676 (+ 153us) server_negotiation.cc:407] Connection header received
0923 23:12:22.595769 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.595770 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.595780 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.595810 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.596486 (+ 676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.597210 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598179 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.598462 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598498 (+ 36us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.600030 (+ 1532us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.600032 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.600032 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.600038 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.601580 (+ 1542us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.601934 (+ 354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.601935 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.601935 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.601984 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.602167 (+ 183us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.602168 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.602169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.602308 (+ 139us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.602379 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.602577 (+ 198us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.602655 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:12:22.603261 30379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.595698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35624 (local address 127.16.122.188:43427)
0923 23:12:22.595732 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.595734 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.595934 (+ 200us) server_negotiation.cc:407] Connection header received
0923 23:12:22.596026 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.596026 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.596037 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.596065 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.596627 (+ 562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.597341 (+ 714us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598309 (+ 968us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.598588 (+ 279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598622 (+ 34us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.600067 (+ 1445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.600068 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.600068 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.600074 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.601789 (+ 1715us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.602107 (+ 318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.602108 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.602109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.602155 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.602502 (+ 347us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.602503 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.602504 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.602668 (+ 164us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.602732 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.603121 (+ 389us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.603201 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:12:22.607374 30303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.595480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38538 (local address 127.16.122.189:36635)
0923 23:12:22.595519 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.595521 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.595787 (+ 266us) server_negotiation.cc:407] Connection header received
0923 23:12:22.595897 (+ 110us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.595898 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.595908 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.595940 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.596597 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.597253 (+ 656us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598355 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.598614 (+ 259us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.598655 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.601551 (+ 2896us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.601552 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.601553 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.601560 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.605816 (+ 4256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.606234 (+ 418us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.606236 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.606236 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.606299 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.606509 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.606510 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.606511 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.606640 (+ 129us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.606708 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.606958 (+ 250us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.607279 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:12:22.609117 30226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.602269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35630)
0923 23:12:22.602464 (+ 195us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.602476 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.602524 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.602779 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.602780 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.602940 (+ 160us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.602979 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.602981 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.604154 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.604155 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.604871 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.604874 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.605276 (+ 402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.605277 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.605333 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.605680 (+ 347us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.605684 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.606762 (+ 1078us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.608245 (+ 1483us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.608246 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.608246 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.608395 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.608562 (+ 167us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.608563 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.608563 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.608617 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.608815 (+ 198us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.608815 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.608857 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.608923 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.609022 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":175}
I20260923 23:12:22.609113 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.602324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35630 (local address 127.16.122.188:43427)
0923 23:12:22.602365 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.602366 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.602639 (+ 273us) server_negotiation.cc:407] Connection header received
0923 23:12:22.602661 (+ 22us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.602662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.602674 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.602721 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.603364 (+ 643us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.604045 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.604971 (+ 926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.605190 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.605262 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.606837 (+ 1575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.606838 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.606839 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.606845 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.608175 (+ 1330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.608461 (+ 286us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.608462 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.608462 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.608512 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.608665 (+ 153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.608665 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.608666 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.608765 (+ 99us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.608816 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.608944 (+ 128us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.609022 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:12:22.610965 30224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: Bootstrap starting.
I20260923 23:12:22.611599 30224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:22.611855 30224 log.cc:824] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:22.612653 30224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: No bootstrap required, opened a new log
I20260923 23:12:22.614038 30224 raft_consensus.cc:348] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.614156 30224 raft_consensus.cc:374] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:22.614187 30224 raft_consensus.cc:729] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5ee080ab63b4e02a9d3b6de49f8a70d, State: Initialized, Role: FOLLOWER
I20260923 23:12:22.614310 30224 consensus_queue.cc:260] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.614614 30224 sys_catalog.cc:564] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:22.614604 30402 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.614715 30402 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:12:22.616295 30413 catalog_manager.cc:1558] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:12:22.616398 30413 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:12:22.617602 30156 master_runner.cc:186] Error getting master registration for 127.16.122.189:36635: OK, instance_id { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" instance_seqno: 1790205142411202 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:12:22.619863 30156 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:12:22.642220 30299 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } attempt: 3
Sep 23 23:12:22 dist-test-slave-95hh krb5kdc[30143](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205142, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:12:22.650406 30378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.642607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35642 (local address 127.16.122.188:43427)
0923 23:12:22.642652 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:22.642654 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:22.642690 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:12:22.642757 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:22.642758 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:22.642773 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:22.642851 (+ 78us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:22.643240 (+ 389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.644069 (+ 829us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.644993 (+ 924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.645300 (+ 307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.645376 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.647617 (+ 2241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:22.647619 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:22.647619 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:22.647627 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:22.649256 (+ 1629us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.649639 (+ 383us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.649640 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.649641 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.649700 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:22.649880 (+ 180us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:22.649881 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:22.649881 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:22.650019 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:22.650084 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:22.650227 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:12:22.650309 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:12:22.650406 30301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:22.642575 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:43427 (local address 127.0.0.1:35642)
0923 23:12:22.642644 (+ 69us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:22.642655 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:22.642706 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:22.642867 (+ 161us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:22.642868 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:22.643092 (+ 224us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:22.643173 (+ 81us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.643175 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.644173 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.644174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.644920 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:22.644923 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:22.645373 (+ 450us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:22.645374 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:22.645440 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:22.645926 (+ 486us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:22.645931 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:22.647526 (+ 1595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:22.649343 (+ 1817us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.649344 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.649345 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.649567 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.649761 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:22.649762 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:22.649762 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:22.649825 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:22.650078 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:22.650078 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:22.650128 (+ 50us) client_negotiation.cc:769] Sending connection context
0923 23:12:22.650197 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:12:22.650297 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 23:12:22.652411 30299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: Bootstrap starting.
I20260923 23:12:22.653155 30299 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:22.653471 30299 log.cc:824] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:22.654326 30299 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: No bootstrap required, opened a new log
I20260923 23:12:22.656365 30299 raft_consensus.cc:348] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.657214 30299 raft_consensus.cc:374] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:22.657302 30299 raft_consensus.cc:729] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 567ecfbb4b1f4b08bdc86c3db2070667, State: Initialized, Role: FOLLOWER
I20260923 23:12:22.657466 30299 consensus_queue.cc:260] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.657828 30299 sys_catalog.cc:564] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:22.657792 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.658006 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:12:22.659133 30252 tablet.cc:2369] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:12:22.659794 30427 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:12:22.659940 30427 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:12:22.661836 30231 master_runner.cc:263] No leader master found from master 567ecfbb4b1f4b08bdc86c3db2070667
I20260923 23:12:22.661885 30231 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260923 23:12:22.761330 30177 tablet.cc:2369] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:12:22.803622 30327 tablet.cc:2369] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:12:22.871519 30402 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:22.871649 30402 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.871887 30402 leader_election.cc:288] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 567ecfbb4b1f4b08bdc86c3db2070667 (127.16.122.189:36635), 0181e827f04d4a7aab0bd4ade60e780b (127.16.122.188:43427)
I20260923 23:12:22.872217 30348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0181e827f04d4a7aab0bd4ade60e780b" is_pre_election: true
I20260923 23:12:22.872223 30273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" is_pre_election: true
I20260923 23:12:22.872390 30348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ee080ab63b4e02a9d3b6de49f8a70d in term 0.
I20260923 23:12:22.872388 30273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5ee080ab63b4e02a9d3b6de49f8a70d in term 0.
I20260923 23:12:22.872566 30174 leader_election.cc:302] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0181e827f04d4a7aab0bd4ade60e780b, c5ee080ab63b4e02a9d3b6de49f8a70d; no voters:
I20260923 23:12:22.872699 30402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:12:22.872754 30402 raft_consensus.cc:482] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:22.872776 30402 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:22.873517 30402 raft_consensus.cc:504] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.873641 30402 leader_election.cc:288] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [CANDIDATE]: Term 1 election: Requested vote from peers 567ecfbb4b1f4b08bdc86c3db2070667 (127.16.122.189:36635), 0181e827f04d4a7aab0bd4ade60e780b (127.16.122.188:43427)
I20260923 23:12:22.873801 30273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "567ecfbb4b1f4b08bdc86c3db2070667"
I20260923 23:12:22.873821 30348 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0181e827f04d4a7aab0bd4ade60e780b"
I20260923 23:12:22.873890 30273 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:22.873889 30348 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:22.874754 30273 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee080ab63b4e02a9d3b6de49f8a70d in term 1.
I20260923 23:12:22.874754 30348 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5ee080ab63b4e02a9d3b6de49f8a70d in term 1.
I20260923 23:12:22.874938 30174 leader_election.cc:302] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0181e827f04d4a7aab0bd4ade60e780b, c5ee080ab63b4e02a9d3b6de49f8a70d; no voters:
I20260923 23:12:22.875068 30402 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:22.875283 30402 raft_consensus.cc:686] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [term 1 LEADER]: Becoming Leader. State: Replica: c5ee080ab63b4e02a9d3b6de49f8a70d, State: Running, Role: LEADER
I20260923 23:12:22.875445 30402 consensus_queue.cc:237] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } }
I20260923 23:12:22.875901 30429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ee080ab63b4e02a9d3b6de49f8a70d. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.875984 30429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:22.876169 30431 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:22.876366 30431 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:22.877770 30273 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [term 1 FOLLOWER]: Refusing update from remote peer c5ee080ab63b4e02a9d3b6de49f8a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:22.877861 30348 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [term 1 FOLLOWER]: Refusing update from remote peer c5ee080ab63b4e02a9d3b6de49f8a70d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:22.878180 30429 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:22.878304 30402 consensus_queue.cc:1035] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [LEADER]: Connected to new peer: Peer: permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:22.879526 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ee080ab63b4e02a9d3b6de49f8a70d. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.879626 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:12:22.879901 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c5ee080ab63b4e02a9d3b6de49f8a70d. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.879992 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:12:22.880529 30429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.880615 30429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:22.880681 30429 sys_catalog.cc:455] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.880724 30429 sys_catalog.cc:458] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:22.880765 30416 sys_catalog.cc:455] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.880848 30416 sys_catalog.cc:458] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:12:22.880895 30434 mvcc.cc:205] Tried to move back new op lower bound from 7332680265224933376 to 7332680265218547712. Current Snapshot: MvccSnapshot[applied={T|T < 7332680265224933376}]
I20260923 23:12:22.881065 30381 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c5ee080ab63b4e02a9d3b6de49f8a70d" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 39643 } } peers { permanent_uuid: "567ecfbb4b1f4b08bdc86c3db2070667" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36635 } } peers { permanent_uuid: "0181e827f04d4a7aab0bd4ade60e780b" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 43427 } } }
I20260923 23:12:22.881140 30381 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:12:22.881369 30433 mvcc.cc:205] Tried to move back new op lower bound from 7332680265224933376 to 7332680265218547712. Current Snapshot: MvccSnapshot[applied={T|T < 7332680265224933376 or (T in {7332680265224933376})}]
I20260923 23:12:22.881387 30431 catalog_manager.cc:1347] Generated new cluster ID: cc33779b6fd74a39a017bc1dbd73cc3d
I20260923 23:12:22.881438 30431 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:22.891089 30431 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:12:22.891775 30431 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:22.900259 30431 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c5ee080ab63b4e02a9d3b6de49f8a70d: Generated new TSK 0
I20260923 23:12:22.900535 30431 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:12:23.592195 30392 catalog_manager.cc:1259] Loaded cluster ID: cc33779b6fd74a39a017bc1dbd73cc3d
I20260923 23:12:23.592269 30392 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: loading cluster ID for follower catalog manager: success
I20260923 23:12:23.593077 30392 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: acquiring CA information for follower catalog manager: success
I20260923 23:12:23.593374 30392 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0181e827f04d4a7aab0bd4ade60e780b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:12:23.660411 30427 catalog_manager.cc:1259] Loaded cluster ID: cc33779b6fd74a39a017bc1dbd73cc3d
I20260923 23:12:23.660492 30427 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: loading cluster ID for follower catalog manager: success
I20260923 23:12:23.661301 30427 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: acquiring CA information for follower catalog manager: success
I20260923 23:12:23.661619 30427 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 567ecfbb4b1f4b08bdc86c3db2070667: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:12:23.949836 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30156
I20260923 23:12:23.954468 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30231
I20260923 23:12:23.958753 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30306
2026-09-23T23:12:23Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (2876 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:12:24 dist-test-slave-95hh krb5kdc[30447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:12:24 dist-test-slave-95hh krb5kdc[30447](info): set up 2 sockets
Sep 23 23:12:24 dist-test-slave-95hh krb5kdc[30447](info): commencing operation
krb5kdc: starting...
W20260923 23:12:24.906816 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:12:24 dist-test-slave-95hh krb5kdc[30447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:12:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:12:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:12:25.016235 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41643
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:41509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:25.094174 30460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:25.094373 30460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:25.094403 30460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:25.095538 30460 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:25.095578 30460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:25.095598 30460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:25.095613 30460 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:25.095628 30460 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:25.097059 30460 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:41509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30460
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:25.097291 30460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:25.097483 30460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:25.101847 30468 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:25.101882 30466 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:25.101877 30465 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:25.102057 30460 server_base.cc:1034] running on GCE node
I20260923 23:12:25.102257 30460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:25.102494 30460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:25.103649 30460 hybrid_clock.cc:648] HybridClock initialized: now 1790205145103633 us; error 29 us; skew 500 ppm
Sep 23 23:12:25 dist-test-slave-95hh krb5kdc[30447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205145, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:25.106105 30460 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:25.106542 30460 webserver.cc:460] Webserver started at http://127.16.122.190:42929/ using document root <none> and password file <none>
I20260923 23:12:25.106739 30460 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:25.106786 30460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:25.106909 30460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:25.107959 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "32e604cc4f06446596c3dfde05bd6b31"
format_stamp: "Formatted at 2026-09-23 23:12:25 on dist-test-slave-95hh"
server_key: "47406d5727385f80071a90a335a951ca"
server_key_iv: "932f21ad20ec4dd0420bbe3ed2b080cf"
server_key_version: "encryptionkey@0"
I20260923 23:12:25.108280 30460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "32e604cc4f06446596c3dfde05bd6b31"
format_stamp: "Formatted at 2026-09-23 23:12:25 on dist-test-slave-95hh"
server_key: "47406d5727385f80071a90a335a951ca"
server_key_iv: "932f21ad20ec4dd0420bbe3ed2b080cf"
server_key_version: "encryptionkey@0"
I20260923 23:12:25.109483 30460 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:25.110147 30475 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:25.110303 30460 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:25.110394 30460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "32e604cc4f06446596c3dfde05bd6b31"
format_stamp: "Formatted at 2026-09-23 23:12:25 on dist-test-slave-95hh"
server_key: "47406d5727385f80071a90a335a951ca"
server_key_iv: "932f21ad20ec4dd0420bbe3ed2b080cf"
server_key_version: "encryptionkey@0"
I20260923 23:12:25.110458 30460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:25.152735 30460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:25.154505 30460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:25.154645 30460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:25.158474 30460 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41643
I20260923 23:12:25.158512 30527 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41643 every 8 connection(s)
I20260923 23:12:25.158844 30460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:25.159701 30528 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:25.160616 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30460
I20260923 23:12:25.160730 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:12:25.160913 16874 external_mini_cluster.cc:1428] Setting key 6d6a477d0d1275aa2d30ba891f837be0
I20260923 23:12:25.162945 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31: Bootstrap starting.
I20260923 23:12:25.163573 30528 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:25.163837 30528 log.cc:824] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:25.164453 30528 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31: No bootstrap required, opened a new log
I20260923 23:12:25.166210 30528 raft_consensus.cc:348] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } }
I20260923 23:12:25.166348 30528 raft_consensus.cc:374] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:25.166380 30528 raft_consensus.cc:729] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32e604cc4f06446596c3dfde05bd6b31, State: Initialized, Role: FOLLOWER
I20260923 23:12:25.166514 30528 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } }
I20260923 23:12:25.166595 30528 raft_consensus.cc:388] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:25.166625 30528 raft_consensus.cc:482] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:25.166668 30528 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 0 FOLLOWER]: Advancing to term 1
Sep 23 23:12:25 dist-test-slave-95hh krb5kdc[30447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:25.167372 30528 raft_consensus.cc:504] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } }
I20260923 23:12:25.167480 30528 leader_election.cc:302] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32e604cc4f06446596c3dfde05bd6b31; no voters:
I20260923 23:12:25.167637 30528 leader_election.cc:288] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:12:25.167689 30533 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:25.167887 30528 sys_catalog.cc:564] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:25.167909 30533 raft_consensus.cc:686] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [term 1 LEADER]: Becoming Leader. State: Replica: 32e604cc4f06446596c3dfde05bd6b31, State: Running, Role: LEADER
I20260923 23:12:25.167995 30533 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } }
I20260923 23:12:25.168335 30534 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32e604cc4f06446596c3dfde05bd6b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } } }
I20260923 23:12:25.168785 30534 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:25.168327 30533 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32e604cc4f06446596c3dfde05bd6b31. Latest consensus state: current_term: 1 leader_uuid: "32e604cc4f06446596c3dfde05bd6b31" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32e604cc4f06446596c3dfde05bd6b31" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41643 } } }
I20260923 23:12:25.168999 30533 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31 [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:25.169219 30545 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:25.169760 30545 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:25.171455 30545 catalog_manager.cc:1347] Generated new cluster ID: f79c1891071a4d5d9a32f2fcc39f788d
I20260923 23:12:25.171504 30545 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:25.171643 30531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:25.161450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55866 (local address 127.16.122.190:41643)
0923 23:12:25.161607 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:25.161617 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:25.161646 (+ 29us) server_negotiation.cc:407] Connection header received
0923 23:12:25.162071 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:25.162078 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:25.162130 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:25.162249 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:25.162972 (+ 723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:25.163993 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:25.164774 (+ 781us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:25.165202 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:25.165296 (+ 94us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:25.167773 (+ 2477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:25.167778 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:25.167778 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:25.167791 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:25.169956 (+ 2165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:25.170268 (+ 312us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:25.170271 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:25.170271 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:25.170367 (+ 96us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:25.170586 (+ 219us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:25.170587 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:25.170587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:25.170883 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:25.171020 (+ 137us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:25.171173 (+ 153us) server_negotiation.cc:299] Negotiation successful
0923 23:12:25.171419 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109,"thread_start_us":65,"threads_started":1}
I20260923 23:12:25.172425 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:12:25.195730 30545 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:12:25.196288 30545 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:25.198297 30545 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 32e604cc4f06446596c3dfde05bd6b31: Generated new TSK 0
I20260923 23:12:25.198501 30545 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:12:25.226074 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30460
2026-09-23T23:12:25Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (1266 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:12:25 dist-test-slave-95hh krb5kdc[30555](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:12:25 dist-test-slave-95hh krb5kdc[30555](info): set up 2 sockets
Sep 23 23:12:25 dist-test-slave-95hh krb5kdc[30555](info): commencing operation
krb5kdc: starting...
W20260923 23:12:26.166880 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:12:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:12:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:12:26.277920 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:26.356091 30568 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:26.356294 30568 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:26.356314 30568 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:26.357399 30568 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:26.357437 30568 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:26.357450 30568 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:26.357467 30568 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:26.357477 30568 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:26.358856 30568 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30568
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:26.359088 30568 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:26.359277 30568 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:26.363788 30573 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.363924 30568 server_base.cc:1034] running on GCE node
W20260923 23:12:26.363823 30576 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.363811 30574 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.365173 30568 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:26.365463 30568 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:26.366596 30568 hybrid_clock.cc:648] HybridClock initialized: now 1790205146366579 us; error 32 us; skew 500 ppm
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:26.368891 30568 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:26.369318 30568 webserver.cc:460] Webserver started at http://127.16.122.190:46647/ using document root <none> and password file <none>
I20260923 23:12:26.369534 30568 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:26.369581 30568 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:26.369704 30568 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:26.370707 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "de18a1dfbd274e919e4cc4593bd2033e"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "229c11ec4272356ea68e208d4657703b"
server_key_iv: "dfea41eca25ed0a0387bfb1c46bd08e2"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.371026 30568 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "de18a1dfbd274e919e4cc4593bd2033e"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "229c11ec4272356ea68e208d4657703b"
server_key_iv: "dfea41eca25ed0a0387bfb1c46bd08e2"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.372223 30568 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:26.372861 30583 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.373003 30568 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.373083 30568 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "de18a1dfbd274e919e4cc4593bd2033e"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "229c11ec4272356ea68e208d4657703b"
server_key_iv: "dfea41eca25ed0a0387bfb1c46bd08e2"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.373144 30568 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:26.384037 30568 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:26.385833 30568 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:26.385957 30568 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:26.389678 30568 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:40101
I20260923 23:12:26.389712 30635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:40101 every 8 connection(s)
I20260923 23:12:26.390024 30568 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:26.390935 30636 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:26.392138 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30568
I20260923 23:12:26.392222 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:12:26.392388 16874 external_mini_cluster.cc:1428] Setting key 08b63bc668581f448ca40aa76c7d5a11
I20260923 23:12:26.393524 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap starting.
I20260923 23:12:26.394244 30636 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:26.394543 30636 log.cc:824] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:26.395295 30636 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: No bootstrap required, opened a new log
I20260923 23:12:26.397493 30636 raft_consensus.cc:348] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:26.397668 30636 raft_consensus.cc:374] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:26.397714 30636 raft_consensus.cc:729] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Initialized, Role: FOLLOWER
I20260923 23:12:26.397840 30636 consensus_queue.cc:260] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:26.397943 30636 raft_consensus.cc:388] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:26.397981 30636 raft_consensus.cc:482] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:26.398031 30636 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:26.398697 30636 raft_consensus.cc:504] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:26.398787 30636 leader_election.cc:302] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de18a1dfbd274e919e4cc4593bd2033e; no voters:
I20260923 23:12:26.398936 30636 leader_election.cc:288] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:12:26.398967 30641 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Leader election won for term 1
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:26.399138 30641 raft_consensus.cc:686] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 LEADER]: Becoming Leader. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Running, Role: LEADER
I20260923 23:12:26.399205 30636 sys_catalog.cc:564] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:26.399211 30641 consensus_queue.cc:237] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:26.399528 30641 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de18a1dfbd274e919e4cc4593bd2033e. Latest consensus state: current_term: 1 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:26.399588 30641 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:26.399729 30643 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:26.399777 30643 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
W20260923 23:12:26.401089 30657 catalog_manager.cc:1558] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:12:26.401134 30657 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:12:26.401183 30647 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:26.401329 30647 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:26.402732 30647 catalog_manager.cc:1347] Generated new cluster ID: b5197a7380c6420cb319b18540078d87
I20260923 23:12:26.402779 30647 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:26.403426 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.392834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46136 (local address 127.16.122.190:40101)
0923 23:12:26.392999 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.393007 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.393031 (+ 24us) server_negotiation.cc:407] Connection header received
0923 23:12:26.393431 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.393437 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.393484 (+ 47us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.393608 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:26.394414 (+ 806us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.395538 (+ 1124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.396582 (+ 1044us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.396967 (+ 385us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.397035 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.399684 (+ 2649us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:26.399688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:26.399688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:26.399699 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:26.401637 (+ 1938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.402015 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.402018 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.402018 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.402097 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.402263 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.402264 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.402265 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.402543 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:26.402682 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.403093 (+ 411us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.403226 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122,"thread_start_us":78,"threads_started":1}
I20260923 23:12:26.424157 30647 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:12:26.424896 30647 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:26.429297 30647 catalog_manager.cc:5868] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Generated new TSK 0
I20260923 23:12:26.429500 30647 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:12:26.453780 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:26.533375 30662 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:26.533597 30662 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:26.533630 30662 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:26.534806 30662 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:26.534852 30662 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:26.534888 30662 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:26.536407 30662 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30662
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:26.536652 30662 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:26.536870 30662 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:26.541803 30668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.541803 30667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.541998 30662 server_base.cc:1034] running on GCE node
W20260923 23:12:26.541822 30670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.542279 30662 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:26.542555 30662 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:26.543699 30662 hybrid_clock.cc:648] HybridClock initialized: now 1790205146543682 us; error 28 us; skew 500 ppm
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:26.546259 30662 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:26.546733 30662 webserver.cc:460] Webserver started at http://127.16.122.129:45729/ using document root <none> and password file <none>
I20260923 23:12:26.546940 30662 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:26.546991 30662 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:26.547139 30662 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:26.548184 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "49c937ba7404439981a3a5f07baae849"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "5995fab0ccd6fb5342a25194a393d2d0"
server_key_iv: "9aec8cfb17ee71a55e8cabcd1c815fc7"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.548532 30662 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "49c937ba7404439981a3a5f07baae849"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "5995fab0ccd6fb5342a25194a393d2d0"
server_key_iv: "9aec8cfb17ee71a55e8cabcd1c815fc7"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.549798 30662 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:12:26.550455 30677 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.550606 30662 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:12:26.550688 30662 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "49c937ba7404439981a3a5f07baae849"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "5995fab0ccd6fb5342a25194a393d2d0"
server_key_iv: "9aec8cfb17ee71a55e8cabcd1c815fc7"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.550750 30662 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:26.566036 30662 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:26.567870 30662 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:26.568001 30662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:26.568200 30662 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:26.568485 30662 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:26.568517 30662 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.568545 30662 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:26.568568 30662 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.573848 30662 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42327
I20260923 23:12:26.573895 30790 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42327 every 8 connection(s)
I20260923 23:12:26.574187 30662 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:26.578883 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30662
I20260923 23:12:26.579005 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:12:26.579217 16874 external_mini_cluster.cc:1428] Setting key 73bfd09ae6fcd17968887bbe89b9f8fa
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:26.583228 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.574876 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:46951 (local address 127.16.122.190:40101)
0923 23:12:26.574940 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.574942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.575133 (+ 191us) server_negotiation.cc:407] Connection header received
0923 23:12:26.575665 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.575667 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.575678 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.575724 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:26.576627 (+ 903us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.577145 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.578117 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.578326 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.578377 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.580589 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:26.580590 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:26.580590 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:26.580597 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:26.581934 (+ 1337us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.582321 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.582322 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.582322 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.582371 (+ 49us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.582599 (+ 228us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.582599 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.582600 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.582689 (+ 89us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:26.582735 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.583085 (+ 350us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.583163 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:12:26.583403 30793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.574834 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:46951)
0923 23:12:26.574995 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.575054 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.575555 (+ 501us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.575790 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.575793 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.576347 (+ 554us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:26.576545 (+ 198us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.576551 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.577216 (+ 665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.577217 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.578042 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.578044 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.578387 (+ 343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.578388 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.578484 (+ 96us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.578929 (+ 445us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:26.578940 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:26.580507 (+ 1567us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:26.582004 (+ 1497us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.582009 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.582011 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.582247 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.582419 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.582420 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.582420 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.582548 (+ 128us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.582737 (+ 189us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:26.582741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:26.582900 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.583004 (+ 104us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.583184 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":76,"threads_started":1}
I20260923 23:12:26.583995 30791 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:26.584084 30791 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:26.584247 30791 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:26.584769 30600 ts_manager.cc:194] Registered new tserver with Master: 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
I20260923 23:12:26.585683 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:46951
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:12:26.601518 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:26.680667 30796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:26.680866 30796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:26.680887 30796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:26.681989 30796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:26.682030 30796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:26.682055 30796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:26.683476 30796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30796
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:26.683694 30796 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:26.683902 30796 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:26.688514 30801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.688529 30802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.688553 30804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.688786 30796 server_base.cc:1034] running on GCE node
I20260923 23:12:26.688977 30796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:26.689189 30796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:26.690346 30796 hybrid_clock.cc:648] HybridClock initialized: now 1790205146690325 us; error 32 us; skew 500 ppm
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:26.692876 30796 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:26.693301 30796 webserver.cc:460] Webserver started at http://127.16.122.130:39685/ using document root <none> and password file <none>
I20260923 23:12:26.693502 30796 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:26.693547 30796 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:26.693650 30796 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:26.694656 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "0cee14dc7a6345479804db330321a46f"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "256a7061f3287099526d6b031a4acc59"
server_key_iv: "49c41fb773212593d6513a3fecc78a07"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.694972 30796 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "0cee14dc7a6345479804db330321a46f"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "256a7061f3287099526d6b031a4acc59"
server_key_iv: "49c41fb773212593d6513a3fecc78a07"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.696223 30796 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:12:26.696909 30811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.697089 30796 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:12:26.697170 30796 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "0cee14dc7a6345479804db330321a46f"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "256a7061f3287099526d6b031a4acc59"
server_key_iv: "49c41fb773212593d6513a3fecc78a07"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.697221 30796 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:26.712832 30796 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:26.714514 30796 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:26.714618 30796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:26.714792 30796 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:26.715111 30796 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:26.715152 30796 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.715173 30796 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:26.715189 30796 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.720502 30796 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:36271
I20260923 23:12:26.720582 30924 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:36271 every 8 connection(s)
I20260923 23:12:26.720846 30796 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:12:26.726259 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30796
I20260923 23:12:26.726341 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:12:26.726495 16874 external_mini_cluster.cc:1428] Setting key 0f405a4bd9025ab3784741293060e673
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:26.729574 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.721499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:38591 (local address 127.16.122.190:40101)
0923 23:12:26.721544 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.721546 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.721736 (+ 190us) server_negotiation.cc:407] Connection header received
0923 23:12:26.722244 (+ 508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.722246 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.722259 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.722304 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:26.723266 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.723761 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.724730 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.724918 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.724973 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.727115 (+ 2142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:26.727116 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:26.727117 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:26.727125 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:26.728373 (+ 1248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.728719 (+ 346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.728720 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.728720 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.728774 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.728984 (+ 210us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.728985 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.728985 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.729120 (+ 135us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:26.729180 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.729409 (+ 229us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.729487 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:12:26.729723 30927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.721469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:38591)
0923 23:12:26.721623 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.721658 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.722148 (+ 490us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.722387 (+ 239us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.722391 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.722931 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:26.723179 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.723184 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.723842 (+ 658us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.723843 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.724651 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.724653 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.724991 (+ 338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.724992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.725117 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.725587 (+ 470us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:26.725597 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:26.726992 (+ 1395us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:26.728448 (+ 1456us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.728454 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.728456 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.728656 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.728825 (+ 169us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.728826 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.728826 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.728931 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.729173 (+ 242us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:26.729179 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:26.729294 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.729381 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.729496 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":71,"threads_started":1}
I20260923 23:12:26.730129 30925 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:26.730217 30925 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:26.730391 30925 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:26.730738 30600 ts_manager.cc:194] Registered new tserver with Master: 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271)
I20260923 23:12:26.731119 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:38591
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:12:26.749545 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:26.827572 30930 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:26.827766 30930 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:26.827796 30930 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:26.828905 30930 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:26.828944 30930 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:26.828977 30930 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:26.830364 30930 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.30930
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:26.830590 30930 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:26.830789 30930 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:26.835357 30936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.835357 30935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:26.835582 30938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:26.835575 30930 server_base.cc:1034] running on GCE node
I20260923 23:12:26.835817 30930 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:26.836053 30930 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:26.837208 30930 hybrid_clock.cc:648] HybridClock initialized: now 1790205146837196 us; error 27 us; skew 500 ppm
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:26.839597 30930 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:26.840035 30930 webserver.cc:460] Webserver started at http://127.16.122.131:36959/ using document root <none> and password file <none>
I20260923 23:12:26.840233 30930 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:26.840276 30930 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:26.840407 30930 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:26.841387 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "533adad628cf4a1e9098f0cb06137083"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "44af11745e77f8bf1417306960596f9b"
server_key_iv: "cbee9b8436f3345ddf2f570b4a068a80"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.841701 30930 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "533adad628cf4a1e9098f0cb06137083"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "44af11745e77f8bf1417306960596f9b"
server_key_iv: "cbee9b8436f3345ddf2f570b4a068a80"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.842880 30930 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:26.843554 30945 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.843690 30930 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.843770 30930 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "533adad628cf4a1e9098f0cb06137083"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "44af11745e77f8bf1417306960596f9b"
server_key_iv: "cbee9b8436f3345ddf2f570b4a068a80"
server_key_version: "encryptionkey@0"
I20260923 23:12:26.843823 30930 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:26.855495 30930 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:26.857299 30930 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:26.857398 30930 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:26.857566 30930 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:26.857820 30930 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:26.857848 30930 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.857867 30930 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:26.857878 30930 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:26.862862 30930 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39889
I20260923 23:12:26.862947 31058 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39889 every 8 connection(s)
I20260923 23:12:26.863209 30930 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:26.863540 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 30930
I20260923 23:12:26.863633 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:12:26.863802 16874 external_mini_cluster.cc:1428] Setting key 6e853b5e745dd2953e3d1a434a7345b1
Sep 23 23:12:26 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205146, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:26.871564 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.863958 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:36611 (local address 127.16.122.190:40101)
0923 23:12:26.864017 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.864019 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.864181 (+ 162us) server_negotiation.cc:407] Connection header received
0923 23:12:26.864724 (+ 543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.864725 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.864736 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.864785 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:26.865678 (+ 893us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.866168 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.867126 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.867319 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.867370 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.869322 (+ 1952us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:26.869323 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:26.869323 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:26.869331 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:26.870335 (+ 1004us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.870677 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.870678 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.870679 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.870730 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.870938 (+ 208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.870938 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.870939 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.871083 (+ 144us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:26.871144 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.871389 (+ 245us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.871476 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:12:26.871716 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.863903 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:36611)
0923 23:12:26.864065 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.864109 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.864634 (+ 525us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.864854 (+ 220us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.864858 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.865402 (+ 544us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:26.865591 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.865596 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.866242 (+ 646us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.866244 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.867061 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.867064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.867381 (+ 317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.867382 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.867511 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.867994 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:26.868002 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:26.869238 (+ 1236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:26.870401 (+ 1163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.870407 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.870409 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.870614 (+ 205us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.870782 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:26.870783 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:26.870783 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:26.870885 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:26.871136 (+ 251us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:26.871142 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:26.871274 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.871362 (+ 88us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.871511 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":78,"threads_started":1}
I20260923 23:12:26.872118 31059 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:26.872216 31059 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:26.872409 31059 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:26.872730 30600 ts_manager.cc:194] Registered new tserver with Master: 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889)
I20260923 23:12:26.873102 30600 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:36611
I20260923 23:12:26.875519 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:12:26.881614 30639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.876828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46150 (local address 127.16.122.190:40101)
0923 23:12:26.876867 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.876869 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.876994 (+ 125us) server_negotiation.cc:407] Connection header received
0923 23:12:26.877036 (+ 42us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.877037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.877045 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.877086 (+ 41us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:26.877553 (+ 467us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.877993 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.878637 (+ 644us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.878800 (+ 163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.878840 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.879834 (+ 994us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:26.879835 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:26.879835 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:26.879841 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:26.880822 (+ 981us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.881094 (+ 272us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.881095 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.881095 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.881135 (+ 40us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:26.881244 (+ 109us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:26.881244 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:26.881244 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:26.881359 (+ 115us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:26.881422 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.881502 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.881554 (+ 52us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:12:26.882871 30600 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46150:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:12:26.884421 30600 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:12:26.890352 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36271 (local address 127.0.0.1:57222)
0923 23:12:26.886791 (+ 469us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.886799 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.886839 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.887077 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.887081 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.887083 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:26.887152 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.887155 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888118 (+ 963us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.888119 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.889325 (+ 1206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.889328 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.890066 (+ 738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890067 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.890143 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890157 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.890218 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.890273 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":80,"threads_started":1}
I20260923 23:12:26.890370 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57222 (local address 127.16.122.130:36271)
0923 23:12:26.886720 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.886724 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.886848 (+ 124us) server_negotiation.cc:407] Connection header received
0923 23:12:26.886915 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.886917 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.886933 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.887010 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:26.887235 (+ 225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888037 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.889410 (+ 1373us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.889993 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890058 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890116 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.890243 (+ 127us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.890281 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":252,"thread_start_us":74,"threads_started":1}
I20260923 23:12:26.890555 31068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.0.0.1:38796)
0923 23:12:26.886601 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.886643 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.886710 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.887142 (+ 432us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.887143 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.887145 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:26.887200 (+ 55us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.887202 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888448 (+ 1246us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.888449 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.889484 (+ 1035us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.889485 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.890348 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.890410 (+ 61us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890414 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.890440 (+ 26us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.890483 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":83,"threads_started":1}
I20260923 23:12:26.890663 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886528 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39889 (local address 127.0.0.1:44380)
0923 23:12:26.887240 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.887249 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.887294 (+ 45us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.887507 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.887508 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.887511 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:26.887544 (+ 33us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.887546 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888679 (+ 1133us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.888680 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.889584 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.889587 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.890460 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890472 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.890529 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890532 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.890574 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.890608 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":66,"threads_started":1}
I20260923 23:12:26.890878 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.16.122.131:39889)
0923 23:12:26.886770 (+ 142us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.886774 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.887349 (+ 575us) server_negotiation.cc:407] Connection header received
0923 23:12:26.887405 (+ 56us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.887407 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.887422 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.887497 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:26.887625 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888596 (+ 971us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.889670 (+ 1074us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.890325 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890381 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890432 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.890754 (+ 322us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.890799 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":67,"threads_started":1}
I20260923 23:12:26.891223 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.886527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38796 (local address 127.16.122.129:42327)
0923 23:12:26.886961 (+ 434us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.886965 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.886974 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:26.887008 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.887010 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.887025 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.887139 (+ 114us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:26.887280 (+ 141us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.888352 (+ 1072us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.889552 (+ 1200us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.890249 (+ 697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.890784 (+ 535us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.890850 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.890874 (+ 24us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.890918 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":77,"threads_started":1}
I20260923 23:12:26.891415 30859 tablet_service.cc:1467] Processing CreateTablet for tablet e8c467c39fd14f938b8059a3582d80f1 (DEFAULT_TABLE table=test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:26.891549 30993 tablet_service.cc:1467] Processing CreateTablet for tablet e8c467c39fd14f938b8059a3582d80f1 (DEFAULT_TABLE table=test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:26.891804 30859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c467c39fd14f938b8059a3582d80f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:26.891837 30993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c467c39fd14f938b8059a3582d80f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:26.891868 30725 tablet_service.cc:1467] Processing CreateTablet for tablet e8c467c39fd14f938b8059a3582d80f1 (DEFAULT_TABLE table=test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:26.892156 30725 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e8c467c39fd14f938b8059a3582d80f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:26.894430 31073 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Bootstrap starting.
I20260923 23:12:26.895026 31073 tablet_bootstrap.cc:654] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:26.895123 31074 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Bootstrap starting.
I20260923 23:12:26.895598 31073 log.cc:824] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:26.895628 31075 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Bootstrap starting.
I20260923 23:12:26.895746 31074 tablet_bootstrap.cc:654] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:26.896148 31075 tablet_bootstrap.cc:654] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:26.896167 31074 log.cc:824] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:26.896644 31075 log.cc:824] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:26.896778 31074 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: No bootstrap required, opened a new log
I20260923 23:12:26.896829 31074 ts_tablet_manager.cc:1397] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:26.897332 31075 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: No bootstrap required, opened a new log
I20260923 23:12:26.897341 31073 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: No bootstrap required, opened a new log
I20260923 23:12:26.897413 31073 ts_tablet_manager.cc:1397] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:12:26.897413 31075 ts_tablet_manager.cc:1397] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:26.898371 31074 raft_consensus.cc:348] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.898568 31074 raft_consensus.cc:374] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:26.898603 31074 raft_consensus.cc:729] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cee14dc7a6345479804db330321a46f, State: Initialized, Role: FOLLOWER
I20260923 23:12:26.898703 31074 consensus_queue.cc:260] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.898965 31074 ts_tablet_manager.cc:1428] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:12:26.899340 31073 raft_consensus.cc:348] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.899322 31075 raft_consensus.cc:348] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.899477 31073 raft_consensus.cc:374] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:26.899477 31075 raft_consensus.cc:374] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:26.899509 31073 raft_consensus.cc:729] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533adad628cf4a1e9098f0cb06137083, State: Initialized, Role: FOLLOWER
I20260923 23:12:26.899509 31075 raft_consensus.cc:729] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c937ba7404439981a3a5f07baae849, State: Initialized, Role: FOLLOWER
I20260923 23:12:26.899641 31073 consensus_queue.cc:260] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.899739 30925 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
I20260923 23:12:26.900058 31073 ts_tablet_manager.cc:1428] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260923 23:12:26.900125 31059 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
I20260923 23:12:26.900015 31075 consensus_queue.cc:260] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.900681 31075 ts_tablet_manager.cc:1428] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:12:26.900836 30791 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
I20260923 23:12:26.933022 31080 raft_consensus.cc:482] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:26.933147 31080 raft_consensus.cc:504] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.933545 31080 leader_election.cc:288] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327), 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271)
I20260923 23:12:26.936484 31061 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.933733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36271 (local address 127.16.122.131:44303)
0923 23:12:26.933779 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.933791 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.933976 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.934145 (+ 169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.934146 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.934148 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:26.934208 (+ 60us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.934209 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.934804 (+ 595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.934805 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.935718 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.935721 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.936264 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.936265 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.936331 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.936335 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.936381 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.936410 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26,"mutex_wait_us":3}
I20260923 23:12:26.936504 31069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.933825 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:44303 (local address 127.16.122.130:36271)
0923 23:12:26.933871 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.933873 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.933883 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:12:26.934070 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.934071 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.934081 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.934132 (+ 51us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:26.934268 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.934731 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.935797 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.936195 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.936250 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.936257 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.936397 (+ 140us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.936437 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:12:26.936554 31071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.933844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:33797 (local address 127.16.122.129:42327)
0923 23:12:26.933959 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.933961 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.933989 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:12:26.934068 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.934069 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.934078 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.934122 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:26.934259 (+ 137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.934736 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.935866 (+ 1130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.936233 (+ 367us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.936302 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.936310 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.936447 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.936496 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260923 23:12:26.936785 31082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.933735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.16.122.131:33797)
0923 23:12:26.933931 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.933941 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:26.934002 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:26.934126 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:26.934127 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:26.934129 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:26.934203 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.934205 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.934809 (+ 604us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.934810 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.935787 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:26.935789 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.936312 (+ 523us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.936312 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:26.936382 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.936386 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:26.936436 (+ 50us) client_negotiation.cc:240] Negotiation successful
0923 23:12:26.936471 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"mutex_wait_us":15,"thread_start_us":94,"threads_started":1}
I20260923 23:12:26.937072 30879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c467c39fd14f938b8059a3582d80f1" candidate_uuid: "533adad628cf4a1e9098f0cb06137083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cee14dc7a6345479804db330321a46f" is_pre_election: true
I20260923 23:12:26.937098 30745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c467c39fd14f938b8059a3582d80f1" candidate_uuid: "533adad628cf4a1e9098f0cb06137083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849" is_pre_election: true
I20260923 23:12:26.937214 30879 raft_consensus.cc:2454] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533adad628cf4a1e9098f0cb06137083 in term 0.
I20260923 23:12:26.937228 30745 raft_consensus.cc:2454] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 533adad628cf4a1e9098f0cb06137083 in term 0.
I20260923 23:12:26.937428 30948 leader_election.cc:302] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cee14dc7a6345479804db330321a46f, 533adad628cf4a1e9098f0cb06137083; no voters:
I20260923 23:12:26.937537 31080 raft_consensus.cc:2781] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:12:26.937594 31080 raft_consensus.cc:482] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:26.937609 31080 raft_consensus.cc:3037] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:26.938269 31080 raft_consensus.cc:504] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.938473 31080 leader_election.cc:288] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [CANDIDATE]: Term 1 election: Requested vote from peers 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327), 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271)
I20260923 23:12:26.938545 30879 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c467c39fd14f938b8059a3582d80f1" candidate_uuid: "533adad628cf4a1e9098f0cb06137083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0cee14dc7a6345479804db330321a46f"
I20260923 23:12:26.938562 30745 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e8c467c39fd14f938b8059a3582d80f1" candidate_uuid: "533adad628cf4a1e9098f0cb06137083" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849"
I20260923 23:12:26.938616 30879 raft_consensus.cc:3037] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:26.938623 30745 raft_consensus.cc:3037] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:26.939476 30745 raft_consensus.cc:2454] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533adad628cf4a1e9098f0cb06137083 in term 1.
I20260923 23:12:26.939484 30879 raft_consensus.cc:2454] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 533adad628cf4a1e9098f0cb06137083 in term 1.
I20260923 23:12:26.939666 30947 leader_election.cc:302] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 49c937ba7404439981a3a5f07baae849, 533adad628cf4a1e9098f0cb06137083; no voters:
I20260923 23:12:26.939749 31080 raft_consensus.cc:2781] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:26.939921 31080 raft_consensus.cc:686] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 1 LEADER]: Becoming Leader. State: Replica: 533adad628cf4a1e9098f0cb06137083, State: Running, Role: LEADER
I20260923 23:12:26.940006 31080 consensus_queue.cc:237] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:26.940876 30600 catalog_manager.cc:5495] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 reported cstate change: term changed from 0 to 1, leader changed from <none> to 533adad628cf4a1e9098f0cb06137083 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "533adad628cf4a1e9098f0cb06137083" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } health_report { overall_health: UNKNOWN } } }
I20260923 23:12:26.951273 31072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.948966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44394 (local address 127.16.122.131:39889)
0923 23:12:26.949038 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:26.949040 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:26.949047 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:12:26.949094 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:26.949095 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:26.949113 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:26.949181 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:12:26.949296 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.949793 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.950513 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:26.950716 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:26.950770 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:26.950914 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:12:26.951052 (+ 138us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:12:26.951124 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:26.951144 (+ 20us) server_negotiation.cc:299] Negotiation successful
0923 23:12:26.951166 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:12:26.953059 30879 raft_consensus.cc:1261] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Refusing update from remote peer 533adad628cf4a1e9098f0cb06137083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:26.953076 30745 raft_consensus.cc:1261] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Refusing update from remote peer 533adad628cf4a1e9098f0cb06137083: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:26.953426 31083 consensus_queue.cc:1035] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:26.953554 31080 consensus_queue.cc:1035] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:26.955976 31088 mvcc.cc:205] Tried to move back new op lower bound from 7332680281917038592 to 7332680281866956800. Current Snapshot: MvccSnapshot[applied={T|T < 7332680281917038592}]
I20260923 23:12:26.956583 31087 mvcc.cc:205] Tried to move back new op lower bound from 7332680281917038592 to 7332680281866956800. Current Snapshot: MvccSnapshot[applied={T|T < 7332680281917038592}]
I20260923 23:12:26.957866 30600 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:26.957973 30600 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46150:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:26.958688 30600 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Sending DeleteTablet for 3 replicas of tablet e8c467c39fd14f938b8059a3582d80f1
I20260923 23:12:26.959050 30725 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c467c39fd14f938b8059a3582d80f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:26 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38796
I20260923 23:12:26.959080 30859 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c467c39fd14f938b8059a3582d80f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:26 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:57222
I20260923 23:12:26.959117 30993 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c467c39fd14f938b8059a3582d80f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:26 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:44380
I20260923 23:12:26.959268 31096 tablet_replica.cc:330] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: stopping tablet replica
I20260923 23:12:26.959347 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30662
I20260923 23:12:26.959394 31096 raft_consensus.cc:2229] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:26.959466 31097 tablet_replica.cc:330] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: stopping tablet replica
I20260923 23:12:26.959501 31096 raft_consensus.cc:2258] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:26.959621 31097 raft_consensus.cc:2229] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:26.959733 31097 raft_consensus.cc:2258] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:26.959947 31098 tablet_replica.cc:330] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: stopping tablet replica
I20260923 23:12:26.960041 31098 raft_consensus.cc:2229] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:12:26.960135 31098 raft_consensus.cc:2258] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:26.960305 31097 ts_tablet_manager.cc:1905] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:26.960582 31098 ts_tablet_manager.cc:1905] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:26.961935 31097 ts_tablet_manager.cc:1918] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:26.962069 31097 log.cc:1196] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/e8c467c39fd14f938b8059a3582d80f1
I20260923 23:12:26.962167 31098 ts_tablet_manager.cc:1918] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:26.962232 31098 log.cc:1196] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/e8c467c39fd14f938b8059a3582d80f1
I20260923 23:12:26.962477 31098 ts_tablet_manager.cc:1939] T e8c467c39fd14f938b8059a3582d80f1 P 533adad628cf4a1e9098f0cb06137083: Deleting consensus metadata
I20260923 23:12:26.962477 31097 ts_tablet_manager.cc:1939] T e8c467c39fd14f938b8059a3582d80f1 P 0cee14dc7a6345479804db330321a46f: Deleting consensus metadata
I20260923 23:12:26.962845 30585 catalog_manager.cc:4841] TS 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889): tablet e8c467c39fd14f938b8059a3582d80f1 (table test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]) successfully deleted
I20260923 23:12:26.962845 30586 catalog_manager.cc:4841] TS 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271): tablet e8c467c39fd14f938b8059a3582d80f1 (table test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]) successfully deleted
W20260923 23:12:26.965451 30585 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:42327 (error 108)
I20260923 23:12:26.965850 31070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:26.965744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.0.0.1:38804)
0923 23:12:26.965788 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:26.965821 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:12:26.965996 30585 catalog_manager.cc:4584] TS 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet e8c467c39fd14f938b8059a3582d80f1: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42327: connect: Connection refused (error 111)
I20260923 23:12:26.966714 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30796
I20260923 23:12:26.971629 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30930
I20260923 23:12:26.975397 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 30568
I20260923 23:12:26.979938 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--webserver_interface=127.16.122.190
--webserver_port=46647
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:27.057381 31100 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.057598 31100 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.057629 31100 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.058763 31100 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:27.058802 31100 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.058822 31100 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.058837 31100 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:27.058854 31100 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:27.060323 31100 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31100
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.060564 31100 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.060758 31100 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.068678 31106 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.068712 31107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.068712 31109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.068918 31100 server_base.cc:1034] running on GCE node
I20260923 23:12:27.069177 31100 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.069468 31100 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.070603 31100 hybrid_clock.cc:648] HybridClock initialized: now 1790205147070587 us; error 33 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.072983 31100 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:27.073405 31100 webserver.cc:460] Webserver started at http://127.16.122.190:46647/ using document root <none> and password file <none>
I20260923 23:12:27.073596 31100 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.073645 31100 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.074891 31100 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:12:27.075525 31116 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.075663 31100 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.075742 31100 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "de18a1dfbd274e919e4cc4593bd2033e"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "229c11ec4272356ea68e208d4657703b"
server_key_iv: "dfea41eca25ed0a0387bfb1c46bd08e2"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.076133 31100 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.088984 31100 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.091138 31100 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.091253 31100 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.095022 31100 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:40101
I20260923 23:12:27.095074 31168 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:40101 every 8 connection(s)
I20260923 23:12:27.095455 31100 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:27.096982 31169 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:12:27.097594 31169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap starting.
I20260923 23:12:27.104347 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31100
I20260923 23:12:27.104676 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:42327
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=45729
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:27.105808 31169 log.cc:824] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.107939 31169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:27.108212 31169 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap complete.
I20260923 23:12:27.110334 31169 raft_consensus.cc:348] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.110579 31169 raft_consensus.cc:729] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.110723 31169 consensus_queue.cc:260] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.110783 31169 raft_consensus.cc:388] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:27.110828 31169 raft_consensus.cc:482] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:27.110875 31169 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:12:27.111941 31169 raft_consensus.cc:504] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.112031 31169 leader_election.cc:302] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de18a1dfbd274e919e4cc4593bd2033e; no voters:
I20260923 23:12:27.112191 31169 leader_election.cc:288] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:12:27.112241 31173 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:12:27.112351 31173 raft_consensus.cc:686] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 LEADER]: Becoming Leader. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Running, Role: LEADER
I20260923 23:12:27.112444 31173 consensus_queue.cc:237] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.112470 31169 sys_catalog.cc:564] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:27.112761 31173 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de18a1dfbd274e919e4cc4593bd2033e. Latest consensus state: current_term: 2 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:27.112845 31173 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:27.113149 31178 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:27.113291 31174 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:27.113415 31174 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:27.113960 31178 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:27.114058 31178 catalog_manager.cc:1259] Loaded cluster ID: b5197a7380c6420cb319b18540078d87
I20260923 23:12:27.114086 31178 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:27.114301 31178 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260923 23:12:27.117734 31178 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260923 23:12:27.120236 31178 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:27.120457 31178 catalog_manager.cc:5879] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Loaded TSK: 0
I20260923 23:12:27.120682 31178 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:12:27.186061 31171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.186282 31171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.186316 31171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.187500 31171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.187541 31171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.187578 31171 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:27.189018 31171 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:42327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=45729
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31171
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.189252 31171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.189458 31171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.195283 31197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.195343 31199 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.195343 31196 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.195538 31171 server_base.cc:1034] running on GCE node
I20260923 23:12:27.195744 31171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.196056 31171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.197242 31171 hybrid_clock.cc:648] HybridClock initialized: now 1790205147197225 us; error 32 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.200225 31171 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:27.200793 31171 webserver.cc:460] Webserver started at http://127.16.122.129:45729/ using document root <none> and password file <none>
I20260923 23:12:27.201056 31171 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.201136 31171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.202824 31171 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:27.203589 31206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.203760 31171 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.203845 31171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "49c937ba7404439981a3a5f07baae849"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "5995fab0ccd6fb5342a25194a393d2d0"
server_key_iv: "9aec8cfb17ee71a55e8cabcd1c815fc7"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.204197 31171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.217293 31171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.219455 31171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.219584 31171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.219801 31171 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:27.220221 31213 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:12:27.222105 31171 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:12:27.222152 31171 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:12:27.222184 31171 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:12:27.222790 31171 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:12:27.222823 31171 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:12:27.222923 31213 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Bootstrap starting.
I20260923 23:12:27.229357 31171 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42327
I20260923 23:12:27.229528 31320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42327 every 8 connection(s)
I20260923 23:12:27.229769 31171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:27.234002 31213 log.cc:824] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.236505 31213 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:27.236786 31213 tablet_bootstrap.cc:492] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Bootstrap complete.
I20260923 23:12:27.237572 31213 ts_tablet_manager.cc:1397] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260923 23:12:27.238679 31213 raft_consensus.cc:348] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:27.239007 31213 raft_consensus.cc:729] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c937ba7404439981a3a5f07baae849, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.239169 31213 consensus_queue.cc:260] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } } peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } }
I20260923 23:12:27.239460 31213 ts_tablet_manager.cc:1428] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Time spent starting tablet: real 0.002s user 0.004s sys 0.000s
I20260923 23:12:27.239915 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31171
I20260923 23:12:27.240151 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:36271
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=39685
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:27.249955 31323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.230433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:36841)
0923 23:12:27.230629 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.230675 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.231170 (+ 495us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.234768 (+ 3598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.234773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.235163 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:27.235422 (+ 259us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.235428 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.237048 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.237049 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.238289 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.238293 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.239156 (+ 863us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.239157 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.239294 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.245259 (+ 5965us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:27.245269 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:27.246617 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:27.248367 (+ 1750us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.248376 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.248378 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.248614 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.248850 (+ 236us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.248851 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.248851 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.248964 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.249330 (+ 366us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:27.249339 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:27.249480 (+ 141us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.249563 (+ 83us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.249749 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":146,"thread_start_us":98,"threads_started":1}
I20260923 23:12:27.249954 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.230522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:36841 (local address 127.16.122.190:40101)
0923 23:12:27.234251 (+ 3729us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.234262 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.234326 (+ 64us) server_negotiation.cc:407] Connection header received
0923 23:12:27.234481 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.234488 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.234556 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.234711 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:27.235557 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.236938 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.238390 (+ 1452us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.239016 (+ 626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.239604 (+ 588us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.246739 (+ 7135us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:27.246744 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:27.246744 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:27.246756 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:27.248282 (+ 1526us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.248699 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.248702 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.248702 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.248792 (+ 90us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.249029 (+ 237us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.249030 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.249030 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.249266 (+ 236us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:27.249333 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.249640 (+ 307us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.249761 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3672,"thread_start_us":91,"threads_started":1}
I20260923 23:12:27.250779 31321 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:27.250875 31321 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:27.251091 31321 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:27.251739 31133 ts_manager.cc:194] Registered new tserver with Master: 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
I20260923 23:12:27.252774 31133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:36841
I20260923 23:12:27.253561 31321 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:12:27.260483 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.252271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.0.0.1:38814)
0923 23:12:27.252456 (+ 185us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.252500 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.252575 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.252751 (+ 176us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.252754 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.253067 (+ 313us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:27.253125 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.253127 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.255021 (+ 1894us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.255021 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.255612 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.255614 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.256181 (+ 567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.256181 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.256225 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.256648 (+ 423us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:27.256652 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:27.257817 (+ 1165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:27.259564 (+ 1747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.259569 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.259571 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.259744 (+ 173us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.259921 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.259922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.259922 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.259975 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.260235 (+ 260us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:27.260238 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:27.260276 (+ 38us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.260333 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.260413 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":148,"thread_start_us":91,"threads_started":1}
I20260923 23:12:27.260686 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.252344 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38814 (local address 127.16.122.129:42327)
0923 23:12:27.252520 (+ 176us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.252524 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.252553 (+ 29us) server_negotiation.cc:407] Connection header received
0923 23:12:27.252653 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.252655 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.252668 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.252734 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:27.253785 (+ 1051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.254951 (+ 1166us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.255681 (+ 730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.256120 (+ 439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.256187 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.257891 (+ 1704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:27.257898 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:27.257899 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:27.257906 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:27.259496 (+ 1590us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.259809 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.259812 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.259812 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.259870 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.260021 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.260021 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.260022 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.260183 (+ 161us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:27.260256 (+ 73us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.260387 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.260478 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":97,"threads_started":1}
I20260923 23:12:27.261051 31255 tablet_service.cc:1514] Processing DeleteTablet for tablet e8c467c39fd14f938b8059a3582d80f1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:26 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38814
I20260923 23:12:27.261631 31330 tablet_replica.cc:330] stopping tablet replica
I20260923 23:12:27.261725 31330 raft_consensus.cc:2229] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:27.261794 31330 raft_consensus.cc:2258] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:27.262074 31330 ts_tablet_manager.cc:1905] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:27.263986 31330 ts_tablet_manager.cc:1918] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:27.264045 31330 log.cc:1196] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/e8c467c39fd14f938b8059a3582d80f1
I20260923 23:12:27.264240 31330 ts_tablet_manager.cc:1939] T e8c467c39fd14f938b8059a3582d80f1 P 49c937ba7404439981a3a5f07baae849: Deleting consensus metadata
I20260923 23:12:27.264686 31118 catalog_manager.cc:4841] TS 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327): tablet e8c467c39fd14f938b8059a3582d80f1 (table test-table [id=2bd5bcaf4bc1451686e3e92e7b74b7a5]) successfully deleted
W20260923 23:12:27.323493 31327 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.323691 31327 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.323714 31327 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.324795 31327 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.324831 31327 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.324855 31327 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:27.326249 31327 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:36271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=39685
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31327
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.326437 31327 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.326627 31327 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.331643 31338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.331630 31336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.331630 31335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.331650 31327 server_base.cc:1034] running on GCE node
I20260923 23:12:27.332058 31327 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.332305 31327 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.333443 31327 hybrid_clock.cc:648] HybridClock initialized: now 1790205147333428 us; error 33 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.335981 31327 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:27.336390 31327 webserver.cc:460] Webserver started at http://127.16.122.130:39685/ using document root <none> and password file <none>
I20260923 23:12:27.336586 31327 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.336647 31327 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.337977 31327 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:27.338508 31345 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.338670 31327 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:27.338750 31327 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "0cee14dc7a6345479804db330321a46f"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "256a7061f3287099526d6b031a4acc59"
server_key_iv: "49c41fb773212593d6513a3fecc78a07"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.339123 31327 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.367168 31327 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.369336 31327 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.369477 31327 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.369694 31327 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:27.369982 31327 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:27.370015 31327 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.370043 31327 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:27.370065 31327 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.375434 31327 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:36271
I20260923 23:12:27.375478 31458 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:36271 every 8 connection(s)
I20260923 23:12:27.375844 31327 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:27.384373 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.376543 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37223 (local address 127.16.122.190:40101)
0923 23:12:27.376620 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.376622 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.376856 (+ 234us) server_negotiation.cc:407] Connection header received
0923 23:12:27.377388 (+ 532us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.377389 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.377401 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.377440 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:27.378334 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.378827 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.379882 (+ 1055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.380075 (+ 193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.380120 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.382035 (+ 1915us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:27.382036 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:27.382037 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:27.382044 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:27.383357 (+ 1313us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.383683 (+ 326us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.383684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.383684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.383731 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.383901 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.383901 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.383901 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.383996 (+ 95us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:27.384040 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.384239 (+ 199us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.384303 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:12:27.384555 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.376543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:37223)
0923 23:12:27.376704 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.376748 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.377283 (+ 535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.377511 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.377515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.378071 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:27.378268 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.378273 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.378902 (+ 629us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.378903 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.379807 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.379810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.380126 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.380127 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.380220 (+ 93us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.380664 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:27.380672 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:27.381945 (+ 1273us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:27.383424 (+ 1479us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.383429 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.383430 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.383634 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.383774 (+ 140us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.383775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.383775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.383860 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.384036 (+ 176us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:27.384039 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:27.384153 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.384228 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.384343 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":76,"threads_started":1}
I20260923 23:12:27.385030 31459 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:27.385118 31459 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:27.385264 31459 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:27.385270 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31327
I20260923 23:12:27.385501 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:39889
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=36959
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:27.385636 31133 ts_manager.cc:194] Registered new tserver with Master: 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271)
I20260923 23:12:27.386077 31133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:37223
W20260923 23:12:27.477640 31462 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.477846 31462 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.477880 31462 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.479055 31462 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.479099 31462 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.479123 31462 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:27.480559 31462 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:39889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=36959
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31462
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.480747 31462 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.480939 31462 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.485559 31470 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.485560 31467 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.485560 31468 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.485710 31462 server_base.cc:1034] running on GCE node
I20260923 23:12:27.485884 31462 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.486109 31462 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.487234 31462 hybrid_clock.cc:648] HybridClock initialized: now 1790205147487226 us; error 26 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.489475 31462 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:27.489881 31462 webserver.cc:460] Webserver started at http://127.16.122.131:36959/ using document root <none> and password file <none>
I20260923 23:12:27.490090 31462 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.490147 31462 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.491434 31462 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:12:27.491925 31477 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.492074 31462 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.492151 31462 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "533adad628cf4a1e9098f0cb06137083"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "44af11745e77f8bf1417306960596f9b"
server_key_iv: "cbee9b8436f3345ddf2f570b4a068a80"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.492462 31462 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.521684 31462 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.523582 31462 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.523715 31462 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.523934 31462 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:27.524230 31462 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:27.524263 31462 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.524292 31462 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:27.524309 31462 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.529654 31462 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39889
I20260923 23:12:27.529708 31590 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39889 every 8 connection(s)
I20260923 23:12:27.530037 31462 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:27.530333 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31462
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:27.537401 31594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.530747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46154 (local address 127.16.122.190:40101)
0923 23:12:27.530824 (+ 77us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.530826 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.530834 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:12:27.530912 (+ 78us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.530913 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.530925 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.531008 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:27.531578 (+ 570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.532345 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.533316 (+ 971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.533543 (+ 227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.533599 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.534890 (+ 1291us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:27.534891 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:27.534892 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:27.534899 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:27.536304 (+ 1405us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.536670 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.536671 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.536671 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.536726 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.536896 (+ 170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.536897 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.536897 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.537020 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:27.537175 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.537222 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.537303 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67,"thread_start_us":50,"threads_started":1}
I20260923 23:12:27.539434 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.530723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:38127 (local address 127.16.122.190:40101)
0923 23:12:27.530769 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.530771 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.531015 (+ 244us) server_negotiation.cc:407] Connection header received
0923 23:12:27.531542 (+ 527us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.531543 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.531554 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.531606 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:27.532716 (+ 1110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.533402 (+ 686us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.534850 (+ 1448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.535082 (+ 232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.535130 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.537261 (+ 2131us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:27.537262 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:27.537263 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:27.537269 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:27.538443 (+ 1174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.538765 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.538766 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.538766 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.538804 (+ 38us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:27.538954 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:27.538954 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:27.538954 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:27.539060 (+ 106us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:27.539107 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.539321 (+ 214us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.539375 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:12:27.539606 31595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.530708 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:38127)
0923 23:12:27.530885 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.530933 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.531445 (+ 512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.531690 (+ 245us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.531695 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.532408 (+ 713us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:27.532635 (+ 227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.532640 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.533509 (+ 869us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.533510 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.534751 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.534754 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.535146 (+ 392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.535147 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.535286 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.535649 (+ 363us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:27.535658 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:27.537182 (+ 1524us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:27.538508 (+ 1326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.538513 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.538515 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.538715 (+ 200us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.538838 (+ 123us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:27.538838 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:27.538839 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:27.538917 (+ 78us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:27.539110 (+ 193us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:27.539114 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:27.539226 (+ 112us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.539302 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.539414 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":83,"threads_started":1}
I20260923 23:12:27.539984 31591 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40101
I20260923 23:12:27.540072 31591 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:27.540234 31591 heartbeater.cc:511] Master 127.16.122.190:40101 requested a full tablet report, sending...
I20260923 23:12:27.540555 31133 ts_manager.cc:194] Registered new tserver with Master: 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889)
I20260923 23:12:27.540894 31133 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:38127
I20260923 23:12:27.548919 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:12:27.551349 31324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.549264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46168 (local address 127.16.122.190:40101)
0923 23:12:27.549305 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.549307 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.549395 (+ 88us) server_negotiation.cc:407] Connection header received
0923 23:12:27.549457 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.549458 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.549498 (+ 40us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.549553 (+ 55us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:12:27.549640 (+ 87us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.550093 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.550749 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.550916 (+ 167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.550957 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.551117 (+ 160us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:12:27.551212 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:12:27.551269 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.551284 (+ 15us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.551298 (+ 14us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:12:27.551592 31133 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:46168:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:12:27.553125 31133 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:12:27.555220 31255 tablet_service.cc:1467] Processing CreateTablet for tablet 065a9ee321c74abe9d2570c693e640ef (DEFAULT_TABLE table=test-table [id=78dcacf24f9b4c728b6ebc254432153a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:27.555416 31255 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a9ee321c74abe9d2570c693e640ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:27.556865 31213 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Bootstrap starting.
I20260923 23:12:27.557132 31213 tablet_bootstrap.cc:654] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:27.557968 31213 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: No bootstrap required, opened a new log
I20260923 23:12:27.558030 31213 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:27.558166 31213 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.558243 31213 raft_consensus.cc:374] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:27.558292 31213 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c937ba7404439981a3a5f07baae849, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.558343 31213 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.558446 31213 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Time spent starting tablet: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.558580 31328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.554780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39889 (local address 127.0.0.1:44402)
0923 23:12:27.554881 (+ 101us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.554891 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.554921 (+ 30us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.555309 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.555310 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.555312 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:27.555355 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.555357 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.556337 (+ 980us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.556338 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.557453 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.557455 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.558323 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.558324 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.558398 (+ 74us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.558420 (+ 22us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.558452 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.558515 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260923 23:12:27.558877 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.554979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44402 (local address 127.16.122.131:39889)
0923 23:12:27.555159 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.555164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.555173 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:27.555209 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.555211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.555224 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.555294 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:27.555407 (+ 113us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.556248 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.557542 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.558227 (+ 685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.558670 (+ 443us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.558732 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.558757 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.558799 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":146,"thread_start_us":117,"threads_started":1}
I20260923 23:12:27.558943 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.554813 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36271 (local address 127.0.0.1:57236)
0923 23:12:27.554956 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.554965 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.555014 (+ 49us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.555374 (+ 360us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.555375 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.555377 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:27.555430 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.555432 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.556638 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.556639 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.557660 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.557662 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.558718 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.558719 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.558787 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.558790 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.558836 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.558879 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":92,"threads_started":1}
I20260923 23:12:27.558987 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.554934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57236 (local address 127.16.122.130:36271)
0923 23:12:27.555173 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.555178 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.555187 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:27.555223 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.555225 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.555240 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.555519 (+ 279us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:27.555588 (+ 69us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.556539 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.557738 (+ 1199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.558327 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.558429 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.558491 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.558863 (+ 372us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.558906 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212,"thread_start_us":131,"threads_started":1}
I20260923 23:12:27.559553 31525 tablet_service.cc:1467] Processing CreateTablet for tablet 065a9ee321c74abe9d2570c693e640ef (DEFAULT_TABLE table=test-table [id=78dcacf24f9b4c728b6ebc254432153a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:27.559581 31393 tablet_service.cc:1467] Processing CreateTablet for tablet 065a9ee321c74abe9d2570c693e640ef (DEFAULT_TABLE table=test-table [id=78dcacf24f9b4c728b6ebc254432153a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:27.559845 31393 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a9ee321c74abe9d2570c693e640ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:27.559840 31525 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a9ee321c74abe9d2570c693e640ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:27.562281 31601 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Bootstrap starting.
I20260923 23:12:27.562283 31602 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Bootstrap starting.
I20260923 23:12:27.562882 31602 tablet_bootstrap.cc:654] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:27.562899 31601 tablet_bootstrap.cc:654] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:27.563373 31602 log.cc:824] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.563375 31601 log.cc:824] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.564006 31601 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: No bootstrap required, opened a new log
I20260923 23:12:27.564021 31602 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: No bootstrap required, opened a new log
I20260923 23:12:27.564065 31602 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:12:27.564064 31601 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:12:27.565331 31602 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.565331 31601 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.565459 31602 raft_consensus.cc:374] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:27.565459 31601 raft_consensus.cc:374] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:27.565482 31602 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533adad628cf4a1e9098f0cb06137083, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.565496 31601 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cee14dc7a6345479804db330321a46f, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.565583 31602 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.565608 31601 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.565846 31602 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:12:27.565847 31601 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:12:27.565874 31591 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
I20260923 23:12:27.565985 31459 heartbeater.cc:503] Master 127.16.122.190:40101 was elected leader, sending a full tablet report...
I20260923 23:12:27.602926 31606 raft_consensus.cc:482] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:27.603094 31606 raft_consensus.cc:504] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.603482 31606 leader_election.cc:288] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889), 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
I20260923 23:12:27.606118 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.603620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39889 (local address 127.16.122.130:35929)
0923 23:12:27.603675 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.603687 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.603743 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.603900 (+ 157us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.603902 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.603904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:27.603982 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.603983 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.604577 (+ 594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.604578 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.605327 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.605330 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.605876 (+ 546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.605877 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.605958 (+ 81us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.605962 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.606014 (+ 52us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.606056 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 23:12:27.606155 31599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.603710 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:35929 (local address 127.16.122.131:39889)
0923 23:12:27.603770 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.603772 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.603799 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:12:27.603826 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.603827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.603836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.603879 (+ 43us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:27.604043 (+ 164us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.604509 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.605411 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.605809 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.605872 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.605881 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.606037 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.606087 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:12:27.606536 31329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.603809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:44035 (local address 127.16.122.129:42327)
0923 23:12:27.603876 (+ 67us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.603878 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.603966 (+ 88us) server_negotiation.cc:407] Connection header received
0923 23:12:27.604036 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.604037 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.604061 (+ 24us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.604116 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:27.604260 (+ 144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.604768 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.605660 (+ 892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.606136 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.606203 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.606288 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.606423 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.606474 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:12:27.606482 31545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533adad628cf4a1e9098f0cb06137083" is_pre_election: true
I20260923 23:12:27.606614 31607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.603725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.16.122.130:44035)
0923 23:12:27.603907 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.603916 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.603983 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.604122 (+ 139us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:27.604123 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:27.604125 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:27.604203 (+ 78us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.604204 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.604847 (+ 643us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.604848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.605581 (+ 733us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:27.605583 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.606272 (+ 689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.606273 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:27.606346 (+ 73us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.606350 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:27.606406 (+ 56us) client_negotiation.cc:240] Negotiation successful
0923 23:12:27.606444 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":166,"thread_start_us":123,"threads_started":1}
I20260923 23:12:27.606640 31545 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 0.
I20260923 23:12:27.606856 31347 leader_election.cc:302] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cee14dc7a6345479804db330321a46f, 533adad628cf4a1e9098f0cb06137083; no voters:
I20260923 23:12:27.606885 31273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849" is_pre_election: true
I20260923 23:12:27.606982 31606 raft_consensus.cc:2781] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:12:27.607059 31606 raft_consensus.cc:482] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:27.607082 31606 raft_consensus.cc:3037] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:27.607074 31273 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 0.
I20260923 23:12:27.607800 31606 raft_consensus.cc:504] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.607932 31606 leader_election.cc:288] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 1 election: Requested vote from peers 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889), 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
I20260923 23:12:27.608079 31545 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "533adad628cf4a1e9098f0cb06137083"
I20260923 23:12:27.608098 31273 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849"
I20260923 23:12:27.608162 31273 raft_consensus.cc:3037] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:27.608162 31545 raft_consensus.cc:3037] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:27.609004 31273 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 1.
I20260923 23:12:27.609004 31545 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 1.
I20260923 23:12:27.609181 31347 leader_election.cc:302] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0cee14dc7a6345479804db330321a46f, 49c937ba7404439981a3a5f07baae849; no voters:
I20260923 23:12:27.609274 31606 raft_consensus.cc:2781] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:27.609416 31606 raft_consensus.cc:686] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 LEADER]: Becoming Leader. State: Replica: 0cee14dc7a6345479804db330321a46f, State: Running, Role: LEADER
I20260923 23:12:27.609500 31606 consensus_queue.cc:237] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.610381 31132 catalog_manager.cc:5495] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f reported cstate change: term changed from 0 to 1, leader changed from <none> to 0cee14dc7a6345479804db330321a46f (127.16.122.130). New cstate: current_term: 1 leader_uuid: "0cee14dc7a6345479804db330321a46f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } health_report { overall_health: UNKNOWN } } }
I20260923 23:12:27.615602 31598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.613381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57250 (local address 127.16.122.130:36271)
0923 23:12:27.613427 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:27.613429 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:27.613436 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:12:27.613513 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:27.613514 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:27.613532 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:27.613592 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:12:27.613699 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.614187 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.614874 (+ 687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:27.615115 (+ 241us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:27.615164 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:27.615312 (+ 148us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:12:27.615412 (+ 100us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:12:27.615482 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:27.615501 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:12:27.615520 (+ 19us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26}
I20260923 23:12:27.617130 31273 raft_consensus.cc:1261] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Refusing update from remote peer 0cee14dc7a6345479804db330321a46f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:27.617146 31545 raft_consensus.cc:1261] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Refusing update from remote peer 0cee14dc7a6345479804db330321a46f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:27.617390 31606 consensus_queue.cc:1035] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:27.617516 31608 consensus_queue.cc:1035] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:27.619199 31611 mvcc.cc:205] Tried to move back new op lower bound from 7332680284637515776 to 7332680284609089536. Current Snapshot: MvccSnapshot[applied={T|T < 7332680284637515776}]
I20260923 23:12:27.620376 31614 mvcc.cc:205] Tried to move back new op lower bound from 7332680284637515776 to 7332680284609089536. Current Snapshot: MvccSnapshot[applied={T|T < 7332680284637515776}]
I20260923 23:12:27.621572 31132 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:46168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:27.621644 31132 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:46168:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:27.622272 31132 catalog_manager.cc:5782] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Sending DeleteTablet for 3 replicas of tablet 065a9ee321c74abe9d2570c693e640ef
I20260923 23:12:27.622532 31525 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a9ee321c74abe9d2570c693e640ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:27 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:44402
I20260923 23:12:27.622550 31255 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a9ee321c74abe9d2570c693e640ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:27 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38814
I20260923 23:12:27.622634 31330 tablet_replica.cc:330] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: stopping tablet replica
I20260923 23:12:27.622681 31330 raft_consensus.cc:2229] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:27.622720 31330 raft_consensus.cc:2258] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:27.622763 31393 tablet_service.cc:1514] Processing DeleteTablet for tablet 065a9ee321c74abe9d2570c693e640ef with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:27 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:57236
I20260923 23:12:27.623054 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31171
I20260923 23:12:27.623061 31620 tablet_replica.cc:330] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: stopping tablet replica
I20260923 23:12:27.623066 31330 ts_tablet_manager.cc:1905] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:27.623704 31621 tablet_replica.cc:330] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: stopping tablet replica
I20260923 23:12:27.623827 31621 raft_consensus.cc:2229] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:12:27.623947 31621 raft_consensus.cc:2258] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:27.627547 31620 raft_consensus.cc:2229] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:27.627671 31620 raft_consensus.cc:2258] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:27.628067 31620 ts_tablet_manager.cc:1905] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:12:27.628538 31118 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:42327 (error 108)
I20260923 23:12:27.629052 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.628917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.0.0.1:38828)
0923 23:12:27.628972 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.629018 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 23:12:27.629051 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31327
I20260923 23:12:27.629833 31620 ts_tablet_manager.cc:1918] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:27.629902 31620 log.cc:1196] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/065a9ee321c74abe9d2570c693e640ef
I20260923 23:12:27.630172 31620 ts_tablet_manager.cc:1939] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Deleting consensus metadata
W20260923 23:12:27.630775 31118 catalog_manager.cc:4584] TS 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 065a9ee321c74abe9d2570c693e640ef: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42327: connect: Connection refused (error 111)
I20260923 23:12:27.631192 31118 catalog_manager.cc:4841] TS 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889): tablet 065a9ee321c74abe9d2570c693e640ef (table test-table [id=78dcacf24f9b4c728b6ebc254432153a]) successfully deleted
I20260923 23:12:27.633980 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31462
I20260923 23:12:27.634044 31597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.633938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36271 (local address 127.0.0.1:57264)
0923 23:12:27.633987 (+ 49us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.634015 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.130:36271: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:12:27.638100 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31100
I20260923 23:12:27.642644 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--webserver_interface=127.16.122.190
--webserver_port=46647
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40101
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:27.721254 31623 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.721454 31623 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.721477 31623 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.722610 31623 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:27.722649 31623 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.722662 31623 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.722672 31623 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:27.722683 31623 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:27.724169 31623 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40101
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40101
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=46647
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31623
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.724395 31623 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.724601 31623 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.729087 31628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.729125 31631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.729117 31629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.729292 31623 server_base.cc:1034] running on GCE node
I20260923 23:12:27.729550 31623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.729815 31623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.730952 31623 hybrid_clock.cc:648] HybridClock initialized: now 1790205147730939 us; error 31 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.733186 31623 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:27.733608 31623 webserver.cc:460] Webserver started at http://127.16.122.190:46647/ using document root <none> and password file <none>
I20260923 23:12:27.733819 31623 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.733865 31623 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.735117 31623 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:27.735723 31638 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.735867 31623 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:27.735946 31623 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "de18a1dfbd274e919e4cc4593bd2033e"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "229c11ec4272356ea68e208d4657703b"
server_key_iv: "dfea41eca25ed0a0387bfb1c46bd08e2"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.736281 31623 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.744354 31623 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.746012 31623 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.746134 31623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.749866 31623 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:40101
I20260923 23:12:27.749898 31690 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:40101 every 8 connection(s)
I20260923 23:12:27.750244 31623 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:27.751752 31691 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:12:27.752358 31691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap starting.
I20260923 23:12:27.757069 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31623
I20260923 23:12:27.757323 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:42327
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=45729
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:27.760355 31691 log.cc:824] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.762496 31691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:27.762763 31691 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e: Bootstrap complete.
I20260923 23:12:27.764833 31691 raft_consensus.cc:348] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.765067 31691 raft_consensus.cc:729] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.765229 31691 consensus_queue.cc:260] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.765307 31691 raft_consensus.cc:388] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:27.765362 31691 raft_consensus.cc:482] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:27.765408 31691 raft_consensus.cc:3037] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:12:27.766383 31691 raft_consensus.cc:504] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.766481 31691 leader_election.cc:302] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: de18a1dfbd274e919e4cc4593bd2033e; no voters:
I20260923 23:12:27.766640 31691 leader_election.cc:288] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 23:12:27.766702 31695 raft_consensus.cc:2781] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 3 FOLLOWER]: Leader election won for term 3
I20260923 23:12:27.766840 31695 raft_consensus.cc:686] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [term 3 LEADER]: Becoming Leader. State: Replica: de18a1dfbd274e919e4cc4593bd2033e, State: Running, Role: LEADER
I20260923 23:12:27.766899 31691 sys_catalog.cc:564] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:27.766963 31695 consensus_queue.cc:237] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } }
I20260923 23:12:27.767282 31695 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: New leader de18a1dfbd274e919e4cc4593bd2033e. Latest consensus state: current_term: 3 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:27.767356 31695 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:27.767282 31696 sys_catalog.cc:455] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "de18a1dfbd274e919e4cc4593bd2033e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "de18a1dfbd274e919e4cc4593bd2033e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40101 } } }
I20260923 23:12:27.767531 31696 sys_catalog.cc:458] T 00000000000000000000000000000000 P de18a1dfbd274e919e4cc4593bd2033e [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:27.767635 31701 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:27.768391 31701 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:27.768497 31701 catalog_manager.cc:1259] Loaded cluster ID: b5197a7380c6420cb319b18540078d87
I20260923 23:12:27.768529 31701 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260923 23:12:27.768740 31701 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f0108f689ad google::LogMessage::Fail() at ??:0
@ 0x7f0108f6b88a google::LogMessage::SendToLog() at ??:0
@ 0x7f0108f684de google::LogMessage::Flush() at ??:0
@ 0x7f0108f69a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f0110e890f1 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f0110e928b9 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f010ae360d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f010ae2b954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f010bc8f6db start_thread at ??:0
@ 0x7f0107a6661f clone at ??:0
W20260923 23:12:27.846793 31693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.847018 31693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.847082 31693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.848263 31693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.848301 31693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.848327 31693 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:27.849848 31693 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:42327
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=45729
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.850087 31693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.850292 31693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.854965 31727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.854978 31728 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.855155 31693 server_base.cc:1034] running on GCE node
W20260923 23:12:27.855245 31730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.855450 31693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.855718 31693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.856886 31693 hybrid_clock.cc:648] HybridClock initialized: now 1790205147856865 us; error 37 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.859732 31693 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:27.860172 31693 webserver.cc:460] Webserver started at http://127.16.122.129:45729/ using document root <none> and password file <none>
I20260923 23:12:27.860400 31693 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.860464 31693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:27.862013 31693 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:27.862681 31737 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:27.862838 31693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:27.862928 31693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "49c937ba7404439981a3a5f07baae849"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "5995fab0ccd6fb5342a25194a393d2d0"
server_key_iv: "9aec8cfb17ee71a55e8cabcd1c815fc7"
server_key_version: "encryptionkey@0"
I20260923 23:12:27.863399 31693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:27.888266 31693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:27.890861 31693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:27.891053 31693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:27.891319 31693 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:27.891834 31744 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:12:27.893286 31693 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:12:27.893347 31693 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:12:27.893379 31693 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:12:27.894307 31693 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:12:27.894354 31693 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:12:27.894443 31744 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Bootstrap starting.
I20260923 23:12:27.896031 31744 log.cc:824] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:27.898813 31744 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:27.899113 31744 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Bootstrap complete.
I20260923 23:12:27.899271 31744 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20260923 23:12:27.900830 31744 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.901111 31693 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42327
I20260923 23:12:27.901167 31744 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49c937ba7404439981a3a5f07baae849, State: Initialized, Role: FOLLOWER
I20260923 23:12:27.901319 31744 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:27.901508 31693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:27.901594 31744 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:12:27.902206 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31693
I20260923 23:12:27.901798 31852 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42327 every 8 connection(s)
I20260923 23:12:27.902496 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:36271
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=39685
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:27.967312 31856 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:12:27.902316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:51081)
0923 23:12:27.902496 (+ 180us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.902846 (+ 350us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:27.903000 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:27.967143 (+ 64143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":82,"threads_started":1}
W20260923 23:12:27.967530 31854 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:40101 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:40101: Client connection negotiation failed: client connection to 127.16.122.190:40101: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:12:27.967900 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.967734 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:51555)
0923 23:12:27.967800 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.967865 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 23:12:27.968220 31856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:27.968142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:47105)
0923 23:12:27.968178 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:27.968203 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260923 23:12:27.968318 31854 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:12:27.986030 31857 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:27.986239 31857 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:27.986260 31857 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:27.987500 31857 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:27.987547 31857 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:27.987576 31857 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:27.989048 31857 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:36271
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=39685
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31857
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:27.989284 31857 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:27.989502 31857 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:27.994336 31862 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.994356 31863 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:27.994365 31865 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:27.994403 31857 server_base.cc:1034] running on GCE node
I20260923 23:12:27.994663 31857 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:27.994863 31857 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:27.995990 31857 hybrid_clock.cc:648] HybridClock initialized: now 1790205147995976 us; error 29 us; skew 500 ppm
Sep 23 23:12:27 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:27.998247 31857 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:27.998646 31857 webserver.cc:460] Webserver started at http://127.16.122.130:39685/ using document root <none> and password file <none>
I20260923 23:12:27.998849 31857 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:27.998893 31857 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:28.000198 31857 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:28.000825 31872 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:28.000960 31857 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:28.001041 31857 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "0cee14dc7a6345479804db330321a46f"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "256a7061f3287099526d6b031a4acc59"
server_key_iv: "49c41fb773212593d6513a3fecc78a07"
server_key_version: "encryptionkey@0"
I20260923 23:12:28.001376 31857 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:28.021166 31857 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:28.022946 31857 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:28.023105 31857 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:28.023326 31857 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:28.023731 31879 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:12:28.025144 31857 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:12:28.025185 31857 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:12:28.025218 31857 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:12:28.025831 31857 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:12:28.025861 31857 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:12:28.025933 31879 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Bootstrap starting.
I20260923 23:12:28.031605 31857 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:36271
I20260923 23:12:28.031694 31986 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:36271 every 8 connection(s)
I20260923 23:12:28.031985 31857 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:12:28.033023 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.032651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:42385)
0923 23:12:28.032811 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.032890 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":71,"threads_started":1}
W20260923 23:12:28.033233 31987 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:40101 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:40101: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
I20260923 23:12:28.033469 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.033382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:38053)
0923 23:12:28.033422 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.033447 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 23:12:28.033767 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.033694 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:54487)
0923 23:12:28.033726 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.033748 (+ 22us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260923 23:12:28.033887 31987 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:12:28.035704 31879 log.cc:824] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:28.037671 31879 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:28.037762 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31857
I20260923 23:12:28.037931 31879 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Bootstrap complete.
I20260923 23:12:28.038023 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:39889
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:40101
--webserver_port=36959
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46511
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:12:28.038767 31879 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20260923 23:12:28.040339 31879 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:28.040592 31879 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0cee14dc7a6345479804db330321a46f, State: Initialized, Role: FOLLOWER
I20260923 23:12:28.040717 31879 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:28.040951 31879 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
I20260923 23:12:28.041193 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.041099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:45167)
0923 23:12:28.041138 (+ 39us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.041167 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:12:28.041463 31989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.041397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:51133)
0923 23:12:28.041432 (+ 35us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.041450 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 23:12:28.118611 31991 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:28.118819 31991 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:28.118842 31991 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:28.120026 31991 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:28.120072 31991 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:28.120098 31991 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:28.121511 31991 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46511
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:39889
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=36959
--tserver_master_addrs=127.16.122.190:40101
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.31991
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:28.121706 31991 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:28.121888 31991 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:28.126606 32000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:28.126595 31998 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:28.126595 31997 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:28.126890 31991 server_base.cc:1034] running on GCE node
I20260923 23:12:28.127110 31991 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:28.127327 31991 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:28.128471 31991 hybrid_clock.cc:648] HybridClock initialized: now 1790205148128454 us; error 32 us; skew 500 ppm
Sep 23 23:12:28 dist-test-slave-95hh krb5kdc[30555](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:28.131011 31991 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:28.131420 31991 webserver.cc:460] Webserver started at http://127.16.122.131:36959/ using document root <none> and password file <none>
I20260923 23:12:28.131603 31991 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:28.131644 31991 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:28.132846 31991 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:28.133401 32007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:28.133551 31991 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:28.133630 31991 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "533adad628cf4a1e9098f0cb06137083"
format_stamp: "Formatted at 2026-09-23 23:12:26 on dist-test-slave-95hh"
server_key: "44af11745e77f8bf1417306960596f9b"
server_key_iv: "cbee9b8436f3345ddf2f570b4a068a80"
server_key_version: "encryptionkey@0"
I20260923 23:12:28.133956 31991 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:28.148051 31991 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:28.149911 31991 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:28.150046 31991 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:28.150261 31991 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:28.150549 31991 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:28.150581 31991 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:28.150609 31991 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:28.150627 31991 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:28.156023 31991 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39889
I20260923 23:12:28.156080 32120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39889 every 8 connection(s)
I20260923 23:12:28.156383 31991 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:28.157346 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.156993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:38937)
0923 23:12:28.157126 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.157204 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":66,"threads_started":1}
W20260923 23:12:28.157691 32121 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:40101 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:40101: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
I20260923 23:12:28.157934 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.157847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:56237)
0923 23:12:28.157891 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.157917 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
I20260923 23:12:28.158146 32123 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.158088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:56707)
0923 23:12:28.158117 (+ 29us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.158135 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":19}
W20260923 23:12:28.158221 32121 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:12:28.163447 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 31991
I20260923 23:12:28.968917 32125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:28.968677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:34767)
0923 23:12:28.968807 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:28.968850 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":64,"threads_started":1}
I20260923 23:12:29.042064 32126 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.041855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:46527)
0923 23:12:29.041967 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.042006 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":59,"threads_started":1}
I20260923 23:12:29.158753 32127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.158546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:41065)
0923 23:12:29.158657 (+ 111us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.158704 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":52,"threads_started":1}
I20260923 23:12:29.643931 32128 raft_consensus.cc:482] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:29.644045 32128 raft_consensus.cc:504] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:29.644344 32128 leader_election.cc:288] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889), 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
Sep 23 23:12:29 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
Sep 23 23:12:29 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.131@KRBTEST.COM
I20260923 23:12:29.655411 32130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.644554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:48821 (local address 127.16.122.131:39889)
0923 23:12:29.644702 (+ 148us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:29.644712 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:29.644752 (+ 40us) server_negotiation.cc:407] Connection header received
0923 23:12:29.645365 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:29.645369 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:29.645415 (+ 46us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:29.645518 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:29.645789 (+ 271us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.647128 (+ 1339us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.648601 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.649232 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.649343 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:29.652122 (+ 2779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:29.652126 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:29.652126 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:29.652136 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:29.653745 (+ 1609us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:29.654250 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:29.654253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:29.654254 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:29.654339 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:29.654596 (+ 257us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:29.654597 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:29.654597 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:29.654838 (+ 241us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:29.654897 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:29.655216 (+ 319us) server_negotiation.cc:299] Negotiation successful
0923 23:12:29.655316 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":49,"threads_started":1}
I20260923 23:12:29.655452 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.644469 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39889 (local address 127.16.122.130:48821)
0923 23:12:29.644569 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.644597 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:29.644793 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:29.645529 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:29.645530 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:29.645653 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:29.645706 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:29.645707 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.647255 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.647256 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:29.648521 (+ 1265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:29.648522 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.649324 (+ 802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.649325 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:29.649454 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:29.649994 (+ 540us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:29.650004 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:29.652030 (+ 2026us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:29.653834 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:29.653840 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:29.653843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:29.654180 (+ 337us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:29.654403 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:29.654404 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:29.654405 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:29.654529 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:29.654899 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:29.654904 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:29.655025 (+ 121us) client_negotiation.cc:769] Sending connection context
0923 23:12:29.655157 (+ 132us) client_negotiation.cc:240] Negotiation successful
0923 23:12:29.655324 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85,"mutex_wait_us":51,"thread_start_us":53,"threads_started":1}
I20260923 23:12:29.655709 32131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.644627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42327 (local address 127.16.122.130:52229)
0923 23:12:29.644716 (+ 89us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.644724 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:29.644802 (+ 78us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:29.645095 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:29.645099 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:29.645363 (+ 264us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:29.645611 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:29.645619 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.647246 (+ 1627us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.647247 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:29.648516 (+ 1269us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:29.648520 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.649326 (+ 806us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.649326 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:29.649454 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:29.650048 (+ 594us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:29.650052 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:29.651639 (+ 1587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:29.653926 (+ 2287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:29.653927 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:29.653927 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:29.654181 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:29.654922 (+ 741us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:29.654923 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:29.654923 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:29.654970 (+ 47us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:29.655409 (+ 439us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:29.655410 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:29.655455 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:12:29.655525 (+ 70us) client_negotiation.cc:240] Negotiation successful
0923 23:12:29.655619 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":80,"mutex_wait_us":77,"thread_start_us":48,"threads_started":1}
I20260923 23:12:29.655810 32132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.644731 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:52229 (local address 127.16.122.129:42327)
0923 23:12:29.644831 (+ 100us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:29.644835 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:29.644843 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:12:29.644930 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:29.644932 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:29.644955 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:29.645063 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:29.645720 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.647128 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.648601 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:29.649232 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:29.649344 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:29.651744 (+ 2400us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:29.651749 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:29.651749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:29.651759 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:29.653838 (+ 2079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:29.654421 (+ 583us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:29.654423 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:29.654424 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:29.654524 (+ 100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:29.655026 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:29.655026 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:29.655027 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:29.655333 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:29.655413 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:29.655602 (+ 189us) server_negotiation.cc:299] Negotiation successful
0923 23:12:29.655722 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75,"thread_start_us":51,"threads_started":1}
W20260923 23:12:29.656317 31874 leader_election.cc:341] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889): Not found: Tablet not found: 065a9ee321c74abe9d2570c693e640ef
I20260923 23:12:29.656318 31807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849" is_pre_election: true
I20260923 23:12:29.656459 31807 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 1.
I20260923 23:12:29.656662 31874 leader_election.cc:302] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cee14dc7a6345479804db330321a46f, 49c937ba7404439981a3a5f07baae849; no voters: 533adad628cf4a1e9098f0cb06137083
I20260923 23:12:29.656766 32128 raft_consensus.cc:2781] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260923 23:12:29.656823 32128 raft_consensus.cc:482] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:29.656841 32128 raft_consensus.cc:3037] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:12:29.657966 32128 raft_consensus.cc:504] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:29.658092 32128 leader_election.cc:288] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 election: Requested vote from peers 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889), 49c937ba7404439981a3a5f07baae849 (127.16.122.129:42327)
I20260923 23:12:29.658303 31807 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "065a9ee321c74abe9d2570c693e640ef" candidate_uuid: "0cee14dc7a6345479804db330321a46f" candidate_term: 2 candidate_status { last_received { term: 1 index: 2 } } ignore_live_leader: false dest_uuid: "49c937ba7404439981a3a5f07baae849"
W20260923 23:12:29.658394 31874 leader_election.cc:341] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 election: Tablet error from VoteRequest() call to peer 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889): Not found: Tablet not found: 065a9ee321c74abe9d2570c693e640ef
I20260923 23:12:29.658414 31807 raft_consensus.cc:3037] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:12:29.659602 31807 raft_consensus.cc:2454] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0cee14dc7a6345479804db330321a46f in term 2.
I20260923 23:12:29.659757 31874 leader_election.cc:302] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0cee14dc7a6345479804db330321a46f, 49c937ba7404439981a3a5f07baae849; no voters: 533adad628cf4a1e9098f0cb06137083
I20260923 23:12:29.659857 32128 raft_consensus.cc:2781] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:12:29.659981 32128 raft_consensus.cc:686] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [term 2 LEADER]: Becoming Leader. State: Replica: 0cee14dc7a6345479804db330321a46f, State: Running, Role: LEADER
I20260923 23:12:29.660070 32128 consensus_queue.cc:237] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 2, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:29.660341 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.660239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:51957)
0923 23:12:29.660280 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.660314 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
I20260923 23:12:29.969590 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:29.969354 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:41411)
0923 23:12:29.969509 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:29.969544 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":90,"threads_started":1}
I20260923 23:12:30.095788 31807 raft_consensus.cc:1261] T 065a9ee321c74abe9d2570c693e640ef P 49c937ba7404439981a3a5f07baae849 [term 2 FOLLOWER]: Refusing update from remote peer 0cee14dc7a6345479804db330321a46f: Log matching property violated. Preceding OpId in replica: term: 1 index: 2. Preceding OpId from leader: term: 2 index: 3. (index mismatch)
I20260923 23:12:30.096125 32128 consensus_queue.cc:1035] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f [LEADER]: Connected to new peer: Peer: permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 3, Last known committed idx: 2, Time since last communication: 0.000s
I20260923 23:12:30.096939 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.096830 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:51047)
0923 23:12:30.096878 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.096908 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
I20260923 23:12:30.097815 32129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.097620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:38615)
0923 23:12:30.097752 (+ 132us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.097784 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260923 23:12:30.097828 32135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.097738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:48153)
0923 23:12:30.097771 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.097798 (+ 27us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":18}
W20260923 23:12:30.097869 31874 consensus_peers.cc:487] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f -> Peer 533adad628cf4a1e9098f0cb06137083 (127.16.122.131:39889): Couldn't send request to peer 533adad628cf4a1e9098f0cb06137083. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 065a9ee321c74abe9d2570c693e640ef. This is attempt 1: this message will repeat every 5th retry.
I20260923 23:12:30.159349 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.159173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:55779)
0923 23:12:30.159278 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.159311 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":60,"threads_started":1}
I20260923 23:12:30.646013 32143 ts_tablet_manager.cc:927] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Initiating tablet copy from peer 0cee14dc7a6345479804db330321a46f (127.16.122.130:36271)
I20260923 23:12:30.646253 32143 tablet_copy_client.cc:316] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: tablet copy: Beginning tablet copy session from remote peer at address 127.16.122.130:36271
Sep 23 23:12:30 dist-test-slave-95hh krb5kdc[30555](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.130@KRBTEST.COM
I20260923 23:12:30.653656 32145 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.646599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40343 (local address 127.16.122.130:36271)
0923 23:12:30.646713 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:30.646717 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:30.646726 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:30.646746 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:30.646747 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:30.646759 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:30.646817 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:30.647304 (+ 487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:30.647987 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:30.648727 (+ 740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:30.649040 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:30.649097 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:30.651182 (+ 2085us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:30.651190 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:30.651191 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:30.651198 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:30.652543 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:30.652941 (+ 398us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:30.652943 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:30.652944 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:30.652994 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:30.653145 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:30.653145 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:30.653146 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:30.653278 (+ 132us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:30.653335 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:30.653522 (+ 187us) server_negotiation.cc:299] Negotiation successful
0923 23:12:30.653586 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82,"thread_start_us":41,"threads_started":1}
I20260923 23:12:30.653671 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.646518 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:36271 (local address 127.16.122.131:40343)
0923 23:12:30.646569 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.646585 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:30.646640 (+ 55us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:30.646825 (+ 185us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:30.646830 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:30.647159 (+ 329us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:30.647230 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:30.647233 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:30.648064 (+ 831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:30.648065 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:30.648652 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:30.648653 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:30.649105 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:30.649107 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:30.649194 (+ 87us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:30.649629 (+ 435us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:30.649633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:30.651091 (+ 1458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:30.652633 (+ 1542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:30.652640 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:30.652642 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:30.652855 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:30.653042 (+ 187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:30.653042 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:30.653043 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:30.653100 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:30.653350 (+ 250us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:30.653357 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:30.653423 (+ 66us) client_negotiation.cc:769] Sending connection context
0923 23:12:30.653495 (+ 72us) client_negotiation.cc:240] Negotiation successful
0923 23:12:30.653588 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 23:12:30.654060 31961 tablet_copy_service.cc:140] P 0cee14dc7a6345479804db330321a46f: Received BeginTabletCopySession request for tablet 065a9ee321c74abe9d2570c693e640ef from peer 533adad628cf4a1e9098f0cb06137083 ({username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:40343)
I20260923 23:12:30.654130 31961 tablet_copy_service.cc:161] P 0cee14dc7a6345479804db330321a46f: Beginning new tablet copy session on tablet 065a9ee321c74abe9d2570c693e640ef from peer 533adad628cf4a1e9098f0cb06137083 at {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:40343: session id = 533adad628cf4a1e9098f0cb06137083-065a9ee321c74abe9d2570c693e640ef
I20260923 23:12:30.654646 31961 tablet_copy_source_session.cc:205] T 065a9ee321c74abe9d2570c693e640ef P 0cee14dc7a6345479804db330321a46f: Tablet Copy: opened 0 blocks and 1 log segments
I20260923 23:12:30.655553 32143 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 065a9ee321c74abe9d2570c693e640ef. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:30.658401 32143 tablet_copy_client.cc:791] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: tablet copy: Starting download of 0 data blocks...
I20260923 23:12:30.658558 32143 tablet_copy_client.cc:655] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: tablet copy: Starting download of 1 WAL segments...
I20260923 23:12:30.659369 32143 tablet_copy_client.cc:531] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260923 23:12:30.660025 32143 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Bootstrap starting.
I20260923 23:12:30.661365 32143 log.cc:824] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:30.663858 32143 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Bootstrap replayed 1/1 log segments. Stats: ops{read=3 overwritten=0 applied=3 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:12:30.664072 32143 tablet_bootstrap.cc:492] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Bootstrap complete.
I20260923 23:12:30.664186 32143 ts_tablet_manager.cc:1397] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Time spent bootstrapping tablet: real 0.004s user 0.004s sys 0.000s
I20260923 23:12:30.665114 32143 raft_consensus.cc:348] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:30.665261 32143 raft_consensus.cc:729] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 533adad628cf4a1e9098f0cb06137083, State: Initialized, Role: FOLLOWER
I20260923 23:12:30.665364 32143 consensus_queue.cc:260] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 3, Last appended: 2.3, Last appended by leader: 3, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "533adad628cf4a1e9098f0cb06137083" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39889 } } peers { permanent_uuid: "0cee14dc7a6345479804db330321a46f" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 36271 } } peers { permanent_uuid: "49c937ba7404439981a3a5f07baae849" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42327 } }
I20260923 23:12:30.665628 32143 ts_tablet_manager.cc:1428] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083: Time spent starting tablet: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:30.665904 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.665789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:49475)
0923 23:12:30.665830 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.665859 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:12:30.665946 31961 tablet_copy_service.cc:342] P 0cee14dc7a6345479804db330321a46f: Request end of tablet copy session 533adad628cf4a1e9098f0cb06137083-065a9ee321c74abe9d2570c693e640ef received from {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:40343
I20260923 23:12:30.665998 31961 tablet_copy_service.cc:433] P 0cee14dc7a6345479804db330321a46f: ending tablet copy session 533adad628cf4a1e9098f0cb06137083-065a9ee321c74abe9d2570c693e640ef on tablet 065a9ee321c74abe9d2570c693e640ef with peer 533adad628cf4a1e9098f0cb06137083
I20260923 23:12:30.666205 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:30.666132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:56419)
0923 23:12:30.666160 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:30.666183 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":13}
I20260923 23:12:31.046991 32150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:31.046789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:48097)
0923 23:12:31.046911 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:31.046949 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":73,"threads_started":1}
I20260923 23:12:31.046945 32075 raft_consensus.cc:1203] T 065a9ee321c74abe9d2570c693e640ef P 533adad628cf4a1e9098f0cb06137083 [term 2 FOLLOWER]: Deduplicated request from leader. Original: 1.2->[2.3-2.3] Dedup: 2.3->[]
I20260923 23:12:31.047391 32141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:31.047292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:35873)
0923 23:12:31.047324 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:31.047356 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
I20260923 23:12:31.098402 32151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:31.098215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:35143)
0923 23:12:31.098315 (+ 100us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:31.098355 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260923 23:12:32.047639 32152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:32.047419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:34285)
0923 23:12:32.047556 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:32.047592 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":83,"threads_started":1}
I20260923 23:12:32.048079 32153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:32.047891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:60811)
0923 23:12:32.047996 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:32.048042 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90,"thread_start_us":56,"threads_started":1}
I20260923 23:12:32.099084 32154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:32.098837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:50347)
0923 23:12:32.098964 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:32.099001 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":61,"threads_started":1}
I20260923 23:12:33.048398 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:33.048147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:45937)
0923 23:12:33.048290 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:33.048337 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":65,"threads_started":1}
I20260923 23:12:33.048708 32156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:33.048533 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:52083)
0923 23:12:33.048643 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:33.048672 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":55,"threads_started":1}
I20260923 23:12:33.048789 32155 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:33.048672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:46257)
0923 23:12:33.048718 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:33.048753 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:12:33.099751 32157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:33.099514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:41893)
0923 23:12:33.099668 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:33.099707 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":135,"thread_start_us":76,"threads_started":1}
I20260923 23:12:34.049419 32159 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:34.049206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:40227)
0923 23:12:34.049341 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:34.049373 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":73,"threads_started":1}
I20260923 23:12:34.049448 32158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:34.049206 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:33569)
0923 23:12:34.049330 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:34.049373 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260923 23:12:34.100482 32160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:34.100242 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:38003)
0923 23:12:34.100387 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:34.100436 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":64,"threads_started":1}
I20260923 23:12:35.050124 32162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:35.049871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:44969)
0923 23:12:35.050030 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:35.050077 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":69,"threads_started":1}
I20260923 23:12:35.050112 32161 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:35.049871 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:52167)
0923 23:12:35.050025 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:35.050058 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":71,"threads_started":1}
I20260923 23:12:35.101248 32163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:35.101006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:50941)
0923 23:12:35.101153 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:35.101199 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":68,"threads_started":1}
I20260923 23:12:36.050753 32164 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:36.050537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:46629)
0923 23:12:36.050668 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:36.050707 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":58,"threads_started":1}
I20260923 23:12:36.050757 32165 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:36.050559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:42657)
0923 23:12:36.050681 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:36.050717 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":59,"threads_started":1}
I20260923 23:12:36.101994 32166 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:36.101767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:53471)
0923 23:12:36.101904 (+ 137us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:36.101947 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":81,"threads_started":1}
I20260923 23:12:37.051385 32167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:37.051190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:48651)
0923 23:12:37.051307 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:37.051340 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100,"thread_start_us":59,"threads_started":1}
I20260923 23:12:37.051385 32168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:37.051186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:47029)
0923 23:12:37.051319 (+ 133us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:37.051343 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":67,"threads_started":1}
I20260923 23:12:37.102669 32169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:37.102467 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:45985)
0923 23:12:37.102590 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:37.102621 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":69,"threads_started":1}
I20260923 23:12:38.052022 32170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:38.051823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:49039)
0923 23:12:38.051940 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:38.051977 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":66,"threads_started":1}
I20260923 23:12:38.052078 32171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:38.051863 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:38201)
0923 23:12:38.051992 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:38.052033 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114,"thread_start_us":58,"threads_started":1}
I20260923 23:12:38.103394 32172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:38.103165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:36843)
0923 23:12:38.103301 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:38.103346 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":60,"threads_started":1}
I20260923 23:12:39.052700 32173 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:39.052504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:57235)
0923 23:12:39.052624 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:39.052653 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":59,"threads_started":1}
I20260923 23:12:39.052706 32174 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:39.052508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:50261)
0923 23:12:39.052631 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:39.052665 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":62,"threads_started":1}
I20260923 23:12:39.104099 32175 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:39.103835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:34659)
0923 23:12:39.103997 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:39.104044 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":72,"threads_started":1}
I20260923 23:12:40.053329 32176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:40.053131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:41823)
0923 23:12:40.053250 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:40.053284 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260923 23:12:40.053350 32177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:40.053172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:60113)
0923 23:12:40.053281 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:40.053310 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":54,"threads_started":1}
I20260923 23:12:40.104776 32178 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:40.104530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:43771)
0923 23:12:40.104689 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:40.104729 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":90,"threads_started":1}
I20260923 23:12:41.054029 32179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:41.053814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:40167)
0923 23:12:41.053943 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:41.053983 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":61,"threads_started":1}
I20260923 23:12:41.054035 32180 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:41.053816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:49925)
0923 23:12:41.053959 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:41.053994 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260923 23:12:41.105578 32181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:41.105279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:49599)
0923 23:12:41.105440 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:41.105495 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":63,"threads_started":1}
I20260923 23:12:42.054675 32183 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:42.054477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:38037)
0923 23:12:42.054604 (+ 127us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:42.054630 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":59,"threads_started":1}
I20260923 23:12:42.054675 32182 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:42.054472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:36863)
0923 23:12:42.054594 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:42.054625 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":60,"threads_started":1}
I20260923 23:12:42.106266 32184 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:42.106049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:49303)
0923 23:12:42.106171 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:42.106213 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":65,"threads_started":1}
I20260923 23:12:43.055347 32185 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:43.055133 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.131:46571)
0923 23:12:43.055245 (+ 112us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:43.055291 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96,"thread_start_us":54,"threads_started":1}
I20260923 23:12:43.055413 32186 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:43.055215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.130:59425)
0923 23:12:43.055338 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:43.055370 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":57,"threads_started":1}
I20260923 23:12:43.106994 32187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:43.106791 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40101 (local address 127.16.122.129:55283)
0923 23:12:43.106919 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:43.106950 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:40101: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":74,"threads_started":1}
I20260923 23:12:43.167305 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31693
I20260923 23:12:43.171805 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31857
I20260923 23:12:43.175511 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 31991
2026-09-23T23:12:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (17951 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:12:43 dist-test-slave-95hh krb5kdc[32190](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:12:43 dist-test-slave-95hh krb5kdc[32190](info): set up 2 sockets
Sep 23 23:12:43 dist-test-slave-95hh krb5kdc[32190](info): commencing operation
krb5kdc: starting...
W20260923 23:12:44.122788 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:12:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:12:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:12:44.234280 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:44.314108 32203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:44.314342 32203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:44.314376 32203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:44.315600 32203 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:12:44.315639 32203 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:12:44.315655 32203 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:12:44.315670 32203 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:12:44.315686 32203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:44.315702 32203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:44.315719 32203 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:12:44.315742 32203 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:12:44.317276 32203 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32203
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:44.317528 32203 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:44.317747 32203 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:44.322779 32208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:44.322796 32203 server_base.cc:1034] running on GCE node
W20260923 23:12:44.322764 32211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.322752 32209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:44.323290 32203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:44.323557 32203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:44.324724 32203 hybrid_clock.cc:648] HybridClock initialized: now 1790205164324721 us; error 49 us; skew 500 ppm
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:44.327025 32203 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:12:44.327505 32203 webserver.cc:460] Webserver started at http://127.16.122.190:43563/ using document root <none> and password file <none>
I20260923 23:12:44.327731 32203 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:44.327780 32203 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:44.327899 32203 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:44.328946 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "1157dc839a93cbf938e9e9e0ae74bfe5"
server_key_iv: "c51effb824876359d594021e4765657e"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.329275 32203 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "1157dc839a93cbf938e9e9e0ae74bfe5"
server_key_iv: "c51effb824876359d594021e4765657e"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.330597 32203 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:12:44.331422 32218 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.331619 32203 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.000s
I20260923 23:12:44.331722 32203 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "1157dc839a93cbf938e9e9e0ae74bfe5"
server_key_iv: "c51effb824876359d594021e4765657e"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.331795 32203 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:44.342583 32203 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:44.344522 32203 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:44.344671 32203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:44.348503 32203 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41679
I20260923 23:12:44.348546 32270 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41679 every 8 connection(s)
I20260923 23:12:44.348886 32203 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:12:44.349011 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 32203
I20260923 23:12:44.349116 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:12:44.349316 16874 external_mini_cluster.cc:1428] Setting key 3b7df6a9b0b9e1d312c3c3ca845e95cf
I20260923 23:12:44.349929 32271 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:44.353020 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap starting.
I20260923 23:12:44.353606 32271 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:44.353917 32271 log.cc:824] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:44.354637 32271 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: No bootstrap required, opened a new log
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:44.356913 32271 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:12:44.357093 32271 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:44.357132 32271 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Initialized, Role: FOLLOWER
I20260923 23:12:44.357265 32271 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:12:44.357342 32271 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:12:44.357372 32271 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:12:44.357419 32271 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:44.358101 32271 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:12:44.358208 32271 leader_election.cc:302] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6803b37ab8ef4b0da1e3b61b9ebb3f23; no voters:
I20260923 23:12:44.358390 32271 leader_election.cc:288] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:12:44.358443 32276 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:44.358651 32271 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:12:44.358660 32276 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 LEADER]: Becoming Leader. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Running, Role: LEADER
I20260923 23:12:44.358770 32276 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:12:44.359146 32277 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:12:44.359180 32278 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6803b37ab8ef4b0da1e3b61b9ebb3f23. Latest consensus state: current_term: 1 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:12:44.359246 32277 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:44.359279 32278 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:12:44.359447 32285 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:12:44.359923 32285 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:12:44.360698 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.349818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36006 (local address 127.16.122.190:41679)
0923 23:12:44.350005 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.350015 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.350048 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:12:44.350584 (+ 536us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.350594 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.350658 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.350803 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:44.351425 (+ 622us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.352732 (+ 1307us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.353555 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.354177 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.354267 (+ 90us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.356828 (+ 2561us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:44.356833 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:44.356833 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:44.356846 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:44.358733 (+ 1887us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.359350 (+ 617us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.359353 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.359354 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.359459 (+ 105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.359713 (+ 254us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.359714 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.359714 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.360023 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:44.360163 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.360384 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.360488 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":92,"threads_started":1}
I20260923 23:12:44.361518 32285 catalog_manager.cc:1347] Generated new cluster ID: a366cf80f260477186b90ec8e6702566
I20260923 23:12:44.361568 32285 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:12:44.370900 32285 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:12:44.371652 32285 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:12:44.377175 32285 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Generated new TSK 0
I20260923 23:12:44.377353 32285 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:12:44.405637 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:44.487622 32297 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:44.487834 32297 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:44.487865 32297 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:44.489024 32297 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:44.489068 32297 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:44.489101 32297 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:12:44.490607 32297 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32297
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:44.490839 32297 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:44.491088 32297 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:44.495904 32305 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.495934 32303 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.495904 32302 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:44.496179 32297 server_base.cc:1034] running on GCE node
I20260923 23:12:44.496397 32297 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:44.496655 32297 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:44.497831 32297 hybrid_clock.cc:648] HybridClock initialized: now 1790205164497815 us; error 31 us; skew 500 ppm
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:44.500387 32297 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:12:44.500828 32297 webserver.cc:460] Webserver started at http://127.16.122.129:39313/ using document root <none> and password file <none>
I20260923 23:12:44.501026 32297 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:44.501076 32297 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:44.501219 32297 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:44.502277 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "1defe561276f45bab2b8971a7a0f4a1e"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "26bc18b0898d29c3e5d39bd82f238ac8"
server_key_iv: "d9c3bdef2893b1f30ed574fce2247387"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.502627 32297 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "1defe561276f45bab2b8971a7a0f4a1e"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "26bc18b0898d29c3e5d39bd82f238ac8"
server_key_iv: "d9c3bdef2893b1f30ed574fce2247387"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.504035 32297 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:12:44.504783 32312 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.504935 32297 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:44.505023 32297 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "1defe561276f45bab2b8971a7a0f4a1e"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "26bc18b0898d29c3e5d39bd82f238ac8"
server_key_iv: "d9c3bdef2893b1f30ed574fce2247387"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.505102 32297 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:44.519374 32297 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:44.521329 32297 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:44.521462 32297 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:44.521690 32297 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:44.521982 32297 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:44.522014 32297 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.522042 32297 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:44.522066 32297 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.527678 32297 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:43857
I20260923 23:12:44.527758 32425 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:43857 every 8 connection(s)
I20260923 23:12:44.528141 32297 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:12:44.531142 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 32297
I20260923 23:12:44.531260 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:12:44.531492 16874 external_mini_cluster.cc:1428] Setting key 0c96329aa3a703e9cff9b1f20509a0e2
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:44.538136 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.528875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:55609 (local address 127.16.122.190:41679)
0923 23:12:44.528922 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.528924 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.529121 (+ 197us) server_negotiation.cc:407] Connection header received
0923 23:12:44.529612 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.529614 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.529622 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.529707 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:44.530668 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.531422 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.532697 (+ 1275us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.532983 (+ 286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.533053 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.535446 (+ 2393us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:44.535447 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:44.535447 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:44.535455 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:44.536776 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.537240 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.537241 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.537241 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.537288 (+ 47us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.537535 (+ 247us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.537536 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.537536 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.537633 (+ 97us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:44.537677 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.538000 (+ 323us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.538067 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:12:44.538405 32428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.528836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:55609)
0923 23:12:44.529000 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.529043 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.529536 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.529784 (+ 248us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.529787 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.530343 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:44.530552 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.530557 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.531530 (+ 973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.531531 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.532610 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.532613 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.533055 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.533056 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.533153 (+ 97us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.533679 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:44.533712 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:44.535342 (+ 1630us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:44.536857 (+ 1515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.536866 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.536868 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.537170 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.537340 (+ 170us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.537341 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.537341 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.537479 (+ 138us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.537693 (+ 214us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:44.537699 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:44.537874 (+ 175us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.537992 (+ 118us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.538165 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":79,"threads_started":1}
I20260923 23:12:44.538951 32426 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:12:44.539085 32426 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:44.539299 32426 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:12:44.539868 32235 ts_manager.cc:194] Registered new tserver with Master: 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857)
I20260923 23:12:44.540486 32235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:55609
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:12:44.557425 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:44.637761 32431 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:44.637975 32431 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:44.638007 32431 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:44.639173 32431 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:44.639210 32431 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:44.639245 32431 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:12:44.640671 32431 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32431
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:44.640898 32431 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:44.641116 32431 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:44.645740 32439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.645742 32436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.645797 32437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:44.645977 32431 server_base.cc:1034] running on GCE node
I20260923 23:12:44.646185 32431 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:44.646412 32431 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:44.647557 32431 hybrid_clock.cc:648] HybridClock initialized: now 1790205164647545 us; error 26 us; skew 500 ppm
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:44.649897 32431 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:12:44.650283 32431 webserver.cc:460] Webserver started at http://127.16.122.130:40629/ using document root <none> and password file <none>
I20260923 23:12:44.650460 32431 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:44.650497 32431 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:44.650584 32431 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:44.651584 32431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "4c80df2d0c564de3a84527857229c856"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "77f462cf3d2a837ddb3dd46f1c0df5b1"
server_key_iv: "644ff7eef19a4334c612319c3ebd8fd9"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.651923 32431 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "4c80df2d0c564de3a84527857229c856"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "77f462cf3d2a837ddb3dd46f1c0df5b1"
server_key_iv: "644ff7eef19a4334c612319c3ebd8fd9"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.653100 32431 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:12:44.653735 32446 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.653899 32431 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:44.653978 32431 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "4c80df2d0c564de3a84527857229c856"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "77f462cf3d2a837ddb3dd46f1c0df5b1"
server_key_iv: "644ff7eef19a4334c612319c3ebd8fd9"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.654031 32431 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:44.686895 32431 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:44.689008 32431 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:44.689117 32431 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:44.689301 32431 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:44.689617 32431 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:44.689653 32431 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.689675 32431 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:44.689697 32431 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.695235 32431 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:42609
I20260923 23:12:44.695310 32559 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:42609 every 8 connection(s)
I20260923 23:12:44.695588 32431 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:44.702689 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 32431
I20260923 23:12:44.702802 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:12:44.703004 16874 external_mini_cluster.cc:1428] Setting key 5dde48e51700a957f117fe453627df9b
I20260923 23:12:44.704725 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.696215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:56189 (local address 127.16.122.190:41679)
0923 23:12:44.696263 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.696265 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.696428 (+ 163us) server_negotiation.cc:407] Connection header received
0923 23:12:44.696933 (+ 505us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.696934 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.696947 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.696995 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:44.697925 (+ 930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.698419 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.699446 (+ 1027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.699648 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.699709 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.701722 (+ 2013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:44.701724 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:44.701724 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:44.701731 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:44.703161 (+ 1430us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.703588 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.703590 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.703590 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.703659 (+ 69us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.704028 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.704029 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.704030 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.704153 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:44.704223 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.704558 (+ 335us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.704628 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:12:44.705035 32562 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.696181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:56189)
0923 23:12:44.696312 (+ 131us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.696352 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.696838 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.697073 (+ 235us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.697077 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.697632 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:44.697833 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.697838 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.698499 (+ 661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.698500 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.699368 (+ 868us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.699371 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.699721 (+ 350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.699722 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.699845 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.700270 (+ 425us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:44.700278 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:44.701606 (+ 1328us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:44.703248 (+ 1642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.703256 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.703259 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.703498 (+ 239us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.703720 (+ 222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.703721 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.703721 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.703884 (+ 163us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.704222 (+ 338us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:44.704229 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:44.704400 (+ 171us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.704513 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.704733 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":59,"threads_started":1}
I20260923 23:12:44.705538 32560 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:12:44.705649 32560 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:44.705852 32560 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
I20260923 23:12:44.706241 32235 ts_manager.cc:194] Registered new tserver with Master: 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
I20260923 23:12:44.706714 32235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:56189
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:12:44.727986 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:12:44.808509 32565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:12:44.808738 32565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:12:44.808761 32565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:12:44.809940 32565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:12:44.809984 32565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:12:44.810009 32565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:12:44.811496 32565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32565
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:12:44.811712 32565 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:12:44.811915 32565 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:12:44.816663 32573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.816676 32571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:12:44.816704 32570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:12:44.816831 32565 server_base.cc:1034] running on GCE node
I20260923 23:12:44.817111 32565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:12:44.817340 32565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:12:44.818495 32565 hybrid_clock.cc:648] HybridClock initialized: now 1790205164818470 us; error 38 us; skew 500 ppm
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:12:44.820922 32565 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:12:44.821353 32565 webserver.cc:460] Webserver started at http://127.16.122.131:34761/ using document root <none> and password file <none>
I20260923 23:12:44.821552 32565 fs_manager.cc:359] Metadata directory not provided
I20260923 23:12:44.821595 32565 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:12:44.821707 32565 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:12:44.822750 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "5b853e63947049789b1e0da12677cab2"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "f5d1ae7d65d95b5f740e9e2f8c690990"
server_key_iv: "dd3cab27bc6663b09a778fcde84e3829"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.823099 32565 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "5b853e63947049789b1e0da12677cab2"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "f5d1ae7d65d95b5f740e9e2f8c690990"
server_key_iv: "dd3cab27bc6663b09a778fcde84e3829"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.824342 32565 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:12:44.825094 32580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.825251 32565 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:12:44.825330 32565 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "5b853e63947049789b1e0da12677cab2"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "f5d1ae7d65d95b5f740e9e2f8c690990"
server_key_iv: "dd3cab27bc6663b09a778fcde84e3829"
server_key_version: "encryptionkey@0"
I20260923 23:12:44.825389 32565 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:12:44.859076 32565 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:12:44.861377 32565 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:12:44.861515 32565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:12:44.861729 32565 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:12:44.862025 32565 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:12:44.862057 32565 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.862085 32565 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:12:44.862109 32565 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:12:44.867597 32565 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:38627
I20260923 23:12:44.867676 32693 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:38627 every 8 connection(s)
I20260923 23:12:44.867977 32565 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:12:44.873018 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 32565
I20260923 23:12:44.873114 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:12:44.873281 16874 external_mini_cluster.cc:1428] Setting key dffb84574ff371755e24b405a64323ba
Sep 23 23:12:44 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205164, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:12:44.877206 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.868628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:59167 (local address 127.16.122.190:41679)
0923 23:12:44.868704 (+ 76us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.868705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.868902 (+ 197us) server_negotiation.cc:407] Connection header received
0923 23:12:44.869390 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.869391 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.869401 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.869457 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:44.870362 (+ 905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.870877 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.872015 (+ 1138us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.872222 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.872279 (+ 57us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.874528 (+ 2249us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:44.874530 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:44.874530 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:44.874538 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:44.875859 (+ 1321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.876249 (+ 390us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.876250 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.876250 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.876310 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.876531 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.876532 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.876533 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.876660 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:44.876715 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.877001 (+ 286us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.877087 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20260923 23:12:44.877357 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.868625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:59167)
0923 23:12:44.868769 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.868814 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.869297 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.869520 (+ 223us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.869523 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.870070 (+ 547us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:12:44.870275 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.870281 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.870990 (+ 709us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.870991 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.871925 (+ 934us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.871928 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.872318 (+ 390us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.872319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.872436 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.872919 (+ 483us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:12:44.872927 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:12:44.874442 (+ 1515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:12:44.875938 (+ 1496us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.875946 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.875948 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.876182 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.876363 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:12:44.876364 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:12:44.876364 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:12:44.876479 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:12:44.876713 (+ 234us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:12:44.876720 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:12:44.876863 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.876952 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.877121 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":65,"threads_started":1}
I20260923 23:12:44.877799 32694 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:12:44.877892 32694 heartbeater.cc:464] Registering TS with master...
I20260923 23:12:44.878041 32694 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
I20260923 23:12:44.878414 32235 ts_manager.cc:194] Registered new tserver with Master: 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627)
I20260923 23:12:44.878787 32235 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:59167
I20260923 23:12:44.885164 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:12:44.892210 32274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.886472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36018 (local address 127.16.122.190:41679)
0923 23:12:44.886525 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.886527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.886638 (+ 111us) server_negotiation.cc:407] Connection header received
0923 23:12:44.886702 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.886703 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.886713 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.886764 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:12:44.887345 (+ 581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.887829 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.888519 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.888707 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.888768 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.889881 (+ 1113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:12:44.889883 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:12:44.889883 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:12:44.889890 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:12:44.891161 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.891466 (+ 305us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.891467 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.891468 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.891520 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:12:44.891678 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:12:44.891679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:12:44.891679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:12:44.891834 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:12:44.891921 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.892045 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.892120 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":29}
I20260923 23:12:44.893676 32235 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36018:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:12:44.895229 32235 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:12:44.901506 32706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.0.0.1:47598)
0923 23:12:44.897413 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.897450 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.897512 (+ 62us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.897940 (+ 428us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.897943 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.897945 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:44.898020 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.898022 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.899107 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.899109 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.900309 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.900311 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901253 (+ 942us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901254 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.901314 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901318 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.901365 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.901408 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":68,"threads_started":1}
I20260923 23:12:44.901582 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40512 (local address 127.16.122.130:42609)
0923 23:12:44.897998 (+ 599us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.898002 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.898010 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:12:44.898049 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.898051 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.898064 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.898277 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:44.898347 (+ 70us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.899414 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.900352 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901041 (+ 689us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901115 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901169 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.901459 (+ 290us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.901505 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":71,"threads_started":1}
I20260923 23:12:44.901592 32708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:38627 (local address 127.0.0.1:44860)
0923 23:12:44.897807 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.897816 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.897866 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.898036 (+ 170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.898037 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.898040 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:44.898102 (+ 62us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.898103 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.899464 (+ 1361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.899466 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.900540 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.900542 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901374 (+ 832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901375 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.901444 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901447 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.901498 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.901530 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":32,"thread_start_us":64,"threads_started":1}
I20260923 23:12:44.901739 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.0.0.1:40512)
0923 23:12:44.897556 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:44.897566 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:44.897616 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:44.898158 (+ 542us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:44.898158 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:44.898160 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:44.898206 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.898207 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.899526 (+ 1319us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.899527 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.900273 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:44.900275 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901123 (+ 848us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901124 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:44.901209 (+ 85us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901236 (+ 27us) client_negotiation.cc:769] Sending connection context
0923 23:12:44.901334 (+ 98us) client_negotiation.cc:240] Negotiation successful
0923 23:12:44.901392 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"mutex_wait_us":2,"thread_start_us":82,"threads_started":1}
I20260923 23:12:44.901953 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47598 (local address 127.16.122.129:43857)
0923 23:12:44.897771 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.897776 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.897787 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:12:44.897831 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.897833 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.897849 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.897926 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:44.898106 (+ 180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.898984 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.900622 (+ 1638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901134 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901234 (+ 100us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901291 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.901836 (+ 545us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.901880 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260923 23:12:44.902235 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:44.897593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44860 (local address 127.16.122.131:38627)
0923 23:12:44.897756 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:44.897761 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:44.897856 (+ 95us) server_negotiation.cc:407] Connection header received
0923 23:12:44.897940 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:44.897942 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:44.897957 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:44.898042 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:44.898205 (+ 163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.899364 (+ 1159us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.900627 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:44.901306 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:44.901375 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:44.901425 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:44.901509 (+ 84us) server_negotiation.cc:299] Negotiation successful
0923 23:12:44.901549 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131,"thread_start_us":85,"threads_started":1}
I20260923 23:12:44.902597 32494 tablet_service.cc:1467] Processing CreateTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f (DEFAULT_TABLE table=test-table [id=854caf1d509543b09682ae1b4f74885c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:44.902601 32360 tablet_service.cc:1467] Processing CreateTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f (DEFAULT_TABLE table=test-table [id=854caf1d509543b09682ae1b4f74885c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:44.902943 32494 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307a2eecfd6c425d8dac4e07f90a343f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:44.902956 32360 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307a2eecfd6c425d8dac4e07f90a343f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:44.902894 32628 tablet_service.cc:1467] Processing CreateTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f (DEFAULT_TABLE table=test-table [id=854caf1d509543b09682ae1b4f74885c]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:12:44.903225 32628 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 307a2eecfd6c425d8dac4e07f90a343f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:12:44.905606 32711 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Bootstrap starting.
I20260923 23:12:44.906145 32713 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap starting.
I20260923 23:12:44.906526 32711 tablet_bootstrap.cc:654] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:44.906721 32713 tablet_bootstrap.cc:654] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:44.907140 32711 log.cc:824] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:44.907167 32712 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Bootstrap starting.
I20260923 23:12:44.907260 32713 log.cc:824] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:44.907847 32712 tablet_bootstrap.cc:654] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Neither blocks nor log segments found. Creating new log.
I20260923 23:12:44.907972 32713 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: No bootstrap required, opened a new log
I20260923 23:12:44.907972 32711 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: No bootstrap required, opened a new log
I20260923 23:12:44.908041 32711 ts_tablet_manager.cc:1397] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Time spent bootstrapping tablet: real 0.003s user 0.000s sys 0.001s
I20260923 23:12:44.908041 32713 ts_tablet_manager.cc:1397] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:44.908365 32712 log.cc:824] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:12:44.909031 32712 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: No bootstrap required, opened a new log
I20260923 23:12:44.909101 32712 ts_tablet_manager.cc:1397] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:12:44.909964 32713 raft_consensus.cc:348] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.909964 32711 raft_consensus.cc:348] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.910146 32713 raft_consensus.cc:374] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:44.910146 32711 raft_consensus.cc:374] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:44.910183 32711 raft_consensus.cc:729] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c80df2d0c564de3a84527857229c856, State: Initialized, Role: FOLLOWER
I20260923 23:12:44.910183 32713 raft_consensus.cc:729] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defe561276f45bab2b8971a7a0f4a1e, State: Initialized, Role: FOLLOWER
I20260923 23:12:44.910306 32711 consensus_queue.cc:260] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.910306 32713 consensus_queue.cc:260] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.910584 32711 ts_tablet_manager.cc:1428] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:12:44.910584 32713 ts_tablet_manager.cc:1428] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:12:44.910617 32560 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
I20260923 23:12:44.910537 32712 raft_consensus.cc:348] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.910689 32426 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
I20260923 23:12:44.910701 32712 raft_consensus.cc:374] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:12:44.910745 32712 raft_consensus.cc:729] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b853e63947049789b1e0da12677cab2, State: Initialized, Role: FOLLOWER
I20260923 23:12:44.910871 32712 consensus_queue.cc:260] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:44.911165 32712 ts_tablet_manager.cc:1428] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:12:44.911274 32694 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
W20260923 23:12:44.946094 32561 tablet.cc:2369] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:12:45.028918 32427 tablet.cc:2369] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:12:45.118508 32695 tablet.cc:2369] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:12:45.247097 32719 raft_consensus.cc:482] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:12:45.247224 32719 raft_consensus.cc:504] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:45.247587 32719 leader_election.cc:288] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
I20260923 23:12:45.251359 32710 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.247891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:56905 (local address 127.16.122.129:43857)
0923 23:12:45.247940 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:45.247942 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:45.248005 (+ 63us) server_negotiation.cc:407] Connection header received
0923 23:12:45.248064 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:45.248065 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:45.248074 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:45.248118 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:45.248235 (+ 117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.248974 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.250404 (+ 1430us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.250948 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.251074 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:45.251085 (+ 11us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:45.251255 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 23:12:45.251299 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:12:45.251360 32707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.247866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:49049 (local address 127.16.122.130:42609)
0923 23:12:45.247967 (+ 101us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:45.247968 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:45.247977 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:12:45.248007 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:45.248008 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:45.248020 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:45.248082 (+ 62us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:12:45.248235 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.248973 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.250405 (+ 1432us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.250947 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.251015 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:45.251024 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:45.251232 (+ 208us) server_negotiation.cc:299] Negotiation successful
0923 23:12:45.251280 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260923 23:12:45.251377 32696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.247795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.16.122.131:49049)
0923 23:12:45.247871 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:45.247882 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:45.247928 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:45.248092 (+ 164us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:45.248093 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:45.248095 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:45.248169 (+ 74us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:45.248171 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.249039 (+ 868us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.249039 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:45.250321 (+ 1282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:45.250323 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.251077 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.251078 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:45.251166 (+ 88us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:45.251170 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:12:45.251225 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 23:12:45.251269 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58,"mutex_wait_us":56}
I20260923 23:12:45.251385 32720 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.247805 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.16.122.131:56905)
0923 23:12:45.247943 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:45.247952 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:12:45.248002 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:12:45.248126 (+ 124us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:12:45.248126 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:12:45.248127 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:12:45.248181 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:45.248182 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.249040 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.249040 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:45.250319 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:12:45.250322 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.251110 (+ 788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.251111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:12:45.251187 (+ 76us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:45.251189 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:12:45.251237 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:12:45.251269 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":124,"mutex_wait_us":63,"thread_start_us":84,"threads_started":1}
I20260923 23:12:45.251864 32514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307a2eecfd6c425d8dac4e07f90a343f" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c80df2d0c564de3a84527857229c856" is_pre_election: true
I20260923 23:12:45.251912 32380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307a2eecfd6c425d8dac4e07f90a343f" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defe561276f45bab2b8971a7a0f4a1e" is_pre_election: true
I20260923 23:12:45.252007 32514 raft_consensus.cc:2454] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 0.
I20260923 23:12:45.252044 32380 raft_consensus.cc:2454] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 0.
I20260923 23:12:45.252247 32583 leader_election.cc:302] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e, 5b853e63947049789b1e0da12677cab2; no voters:
I20260923 23:12:45.252381 32719 raft_consensus.cc:2781] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:12:45.252440 32719 raft_consensus.cc:482] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:12:45.252463 32719 raft_consensus.cc:3037] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:45.253129 32719 raft_consensus.cc:504] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:45.253253 32719 leader_election.cc:288] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 election: Requested vote from peers 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
I20260923 23:12:45.253422 32380 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307a2eecfd6c425d8dac4e07f90a343f" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defe561276f45bab2b8971a7a0f4a1e"
I20260923 23:12:45.253440 32514 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "307a2eecfd6c425d8dac4e07f90a343f" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c80df2d0c564de3a84527857229c856"
I20260923 23:12:45.253499 32514 raft_consensus.cc:3037] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:45.253499 32380 raft_consensus.cc:3037] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:12:45.254173 32514 raft_consensus.cc:2454] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 1.
I20260923 23:12:45.254177 32380 raft_consensus.cc:2454] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 1.
I20260923 23:12:45.254310 32583 leader_election.cc:302] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e, 5b853e63947049789b1e0da12677cab2; no voters:
I20260923 23:12:45.254402 32719 raft_consensus.cc:2781] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:12:45.254540 32719 raft_consensus.cc:686] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b853e63947049789b1e0da12677cab2, State: Running, Role: LEADER
I20260923 23:12:45.254637 32719 consensus_queue.cc:237] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:12:45.255414 32235 catalog_manager.cc:5495] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b853e63947049789b1e0da12677cab2 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "5b853e63947049789b1e0da12677cab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } health_report { overall_health: HEALTHY } } }
I20260923 23:12:45.342600 32709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.340329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44872 (local address 127.16.122.131:38627)
0923 23:12:45.340388 (+ 59us) server_negotiation.cc:206] Beginning negotiation
0923 23:12:45.340391 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:12:45.340411 (+ 20us) server_negotiation.cc:407] Connection header received
0923 23:12:45.340449 (+ 38us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:12:45.340450 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:12:45.340467 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:12:45.340523 (+ 56us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:12:45.340655 (+ 132us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.341129 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.341885 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:12:45.342077 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:12:45.342140 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:12:45.342287 (+ 147us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:12:45.342390 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:12:45.342459 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:12:45.342494 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:12:45.342517 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:12:45.344463 32380 raft_consensus.cc:1261] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Refusing update from remote peer 5b853e63947049789b1e0da12677cab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:45.344463 32514 raft_consensus.cc:1261] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Refusing update from remote peer 5b853e63947049789b1e0da12677cab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:12:45.344817 32721 consensus_queue.cc:1035] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:45.344940 32719 consensus_queue.cc:1035] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:12:45.347183 32724 mvcc.cc:205] Tried to move back new op lower bound from 7332680357248258048 to 7332680356883382272. Current Snapshot: MvccSnapshot[applied={T|T < 7332680357248258048}]
I20260923 23:12:45.348615 32725 mvcc.cc:205] Tried to move back new op lower bound from 7332680357248258048 to 7332680356883382272. Current Snapshot: MvccSnapshot[applied={T|T < 7332680357248258048 or (T in {7332680357248258048})}]
I20260923 23:12:45.349582 32235 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:45.349680 32235 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:36018:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:12:45.350385 32235 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Sending DeleteTablet for 3 replicas of tablet 307a2eecfd6c425d8dac4e07f90a343f
I20260923 23:12:45.350656 32360 tablet_service.cc:1514] Processing DeleteTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:45 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:47598
I20260923 23:12:45.350731 32494 tablet_service.cc:1514] Processing DeleteTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:45 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40512
I20260923 23:12:45.350788 32628 tablet_service.cc:1514] Processing DeleteTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:45 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:44860
I20260923 23:12:45.350994 32735 tablet_replica.cc:330] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: stopping tablet replica
I20260923 23:12:45.351109 32735 raft_consensus.cc:2229] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:12:45.351205 32735 raft_consensus.cc:2258] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:45.351212 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 32297
I20260923 23:12:45.351603 32735 ts_tablet_manager.cc:1905] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:12:45.353149 32735 ts_tablet_manager.cc:1918] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:12:45.353214 32735 log.cc:1196] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/307a2eecfd6c425d8dac4e07f90a343f
I20260923 23:12:45.353438 32735 ts_tablet_manager.cc:1939] T 307a2eecfd6c425d8dac4e07f90a343f P 5b853e63947049789b1e0da12677cab2: Deleting consensus metadata
I20260923 23:12:45.353780 32220 catalog_manager.cc:4841] TS 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): tablet 307a2eecfd6c425d8dac4e07f90a343f (table test-table [id=854caf1d509543b09682ae1b4f74885c]) successfully deleted
I20260923 23:12:45.356834 32734 tablet_replica.cc:330] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: stopping tablet replica
I20260923 23:12:45.356941 32734 raft_consensus.cc:2229] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:12:45.357059 32734 raft_consensus.cc:2258] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:12:45.357422 32734 ts_tablet_manager.cc:1905] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:12:45.358032 32221 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:43857 (error 108)
I20260923 23:12:45.358268 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 32431
I20260923 23:12:45.358783 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.358380 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.0.0.1:47606)
0923 23:12:45.358719 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:45.358753 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319}
W20260923 23:12:45.359700 32221 catalog_manager.cc:4584] TS 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 307a2eecfd6c425d8dac4e07f90a343f: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43857: connect: Connection refused (error 111)
I20260923 23:12:45.362593 32705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:12:45.362501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.0.0.1:40514)
0923 23:12:45.362537 (+ 36us) negotiation.cc:107] Waiting for socket to connect
0923 23:12:45.362562 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.130:42609: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
I20260923 23:12:45.362637 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 32565
I20260923 23:12:45.368259 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 32203
I20260923 23:13:05.372874 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--webserver_interface=127.16.122.190
--webserver_port=43563
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:05.451969 32737 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:05.452198 32737 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:05.452230 32737 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:05.453377 32737 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:13:05.453416 32737 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:13:05.453433 32737 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:13:05.453447 32737 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:05.453464 32737 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:05.453481 32737 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:05.453506 32737 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:05.453531 32737 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:05.455013 32737 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.32737
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:05.455288 32737 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:05.455493 32737 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:05.463465 32743 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.463514 32737 server_base.cc:1034] running on GCE node
W20260923 23:13:05.463660 32746 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:05.463613 32744 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.463940 32737 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:05.464244 32737 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:05.465384 32737 hybrid_clock.cc:648] HybridClock initialized: now 1790205185465371 us; error 30 us; skew 500 ppm
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:05.467741 32737 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:13:05.468187 32737 webserver.cc:460] Webserver started at http://127.16.122.190:43563/ using document root <none> and password file <none>
I20260923 23:13:05.468394 32737 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:05.468456 32737 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:05.469785 32737 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:05.470357 32753 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.470522 32737 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:05.470606 32737 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "1157dc839a93cbf938e9e9e0ae74bfe5"
server_key_iv: "c51effb824876359d594021e4765657e"
server_key_version: "encryptionkey@0"
I20260923 23:13:05.470992 32737 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:05.489262 32737 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:05.491228 32737 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:05.491361 32737 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:05.495294 32737 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41679
I20260923 23:13:05.495345 337 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41679 every 8 connection(s)
I20260923 23:13:05.495715 32737 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:05.497148 338 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:13:05.497689 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 32737
I20260923 23:13:05.497849 338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap starting.
I20260923 23:13:05.497924 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:43857
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=39313
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:13:05.505918 338 log.cc:824] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:05.507798 338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:13:05.508154 338 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap complete.
I20260923 23:13:05.510491 338 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:05.510711 338 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.510845 338 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:05.510910 338 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:05.510991 338 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:05.511056 338 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 1 FOLLOWER]: Advancing to term 2
I20260923 23:13:05.512024 338 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:05.512118 338 leader_election.cc:302] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6803b37ab8ef4b0da1e3b61b9ebb3f23; no voters:
I20260923 23:13:05.512267 338 leader_election.cc:288] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260923 23:13:05.512313 342 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Leader election won for term 2
I20260923 23:13:05.512454 342 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 LEADER]: Becoming Leader. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Running, Role: LEADER
I20260923 23:13:05.512533 338 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:05.512594 342 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:05.512902 342 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6803b37ab8ef4b0da1e3b61b9ebb3f23. Latest consensus state: current_term: 2 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:13:05.512985 342 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:05.513213 343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:13:05.513300 343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:05.513370 349 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:05.514061 349 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:05.514153 349 catalog_manager.cc:1259] Loaded cluster ID: a366cf80f260477186b90ec8e6702566
I20260923 23:13:05.514184 349 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:05.514947 349 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:05.515101 349 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Loaded TSK: 0
I20260923 23:13:05.529119 349 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Generated new TSK 1
I20260923 23:13:05.529217 349 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:13:05.595207 340 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:05.595402 340 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:05.595424 340 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:05.596547 340 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:05.596587 340 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:05.596612 340 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:05.598016 340 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=39313
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.340
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:05.598217 340 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:05.598425 340 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:05.604300 368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.604406 340 server_base.cc:1034] running on GCE node
W20260923 23:13:05.604287 366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:05.604301 365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.604822 340 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:05.605059 340 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:05.606204 340 hybrid_clock.cc:648] HybridClock initialized: now 1790205185606193 us; error 27 us; skew 500 ppm
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:05.608793 340 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:13:05.609306 340 webserver.cc:460] Webserver started at http://127.16.122.129:39313/ using document root <none> and password file <none>
I20260923 23:13:05.609560 340 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:05.609642 340 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:05.611388 340 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:13:05.612149 375 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.612334 340 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:13:05.612449 340 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "1defe561276f45bab2b8971a7a0f4a1e"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "26bc18b0898d29c3e5d39bd82f238ac8"
server_key_iv: "d9c3bdef2893b1f30ed574fce2247387"
server_key_version: "encryptionkey@0"
I20260923 23:13:05.612891 340 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:05.646106 340 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:05.648654 340 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:05.648821 340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:05.649103 340 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:05.649641 382 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:13:05.651093 340 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:13:05.651152 340 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:13:05.651187 340 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:13:05.652061 340 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:13:05.652108 340 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:13:05.652173 382 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap starting.
I20260923 23:13:05.658715 340 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:43857
I20260923 23:13:05.659238 340 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:05.662395 382 log.cc:824] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:05.662813 489 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:43857 every 8 connection(s)
I20260923 23:13:05.663026 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 340
I20260923 23:13:05.663307 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:42609
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=40629
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:13:05.665059 382 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:13:05.665334 382 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap complete.
I20260923 23:13:05.666445 382 ts_tablet_manager.cc:1397] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Time spent bootstrapping tablet: real 0.014s user 0.000s sys 0.010s
I20260923 23:13:05.667765 382 raft_consensus.cc:348] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:13:05.668071 382 raft_consensus.cc:729] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defe561276f45bab2b8971a7a0f4a1e, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.668197 382 consensus_queue.cc:260] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:13:05.668429 382 ts_tablet_manager.cc:1428] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Time spent starting tablet: real 0.002s user 0.005s sys 0.000s
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:05.673360 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.660571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:55211 (local address 127.16.122.190:41679)
0923 23:13:05.661252 (+ 681us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.661262 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.661311 (+ 49us) server_negotiation.cc:407] Connection header received
0923 23:13:05.661446 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.661452 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.661520 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.661652 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.662573 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.664592 (+ 2019us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.666096 (+ 1504us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.666647 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.666753 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.669495 (+ 2742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.669499 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.669500 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.669518 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.671415 (+ 1897us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.671913 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.671916 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.671917 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.672025 (+ 108us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.672303 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.672304 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.672304 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.672579 (+ 275us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.672653 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.673032 (+ 379us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.673166 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":628,"thread_start_us":98,"threads_started":1}
I20260923 23:13:05.673367 492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.660181 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:55211)
0923 23:13:05.660391 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.660439 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.661039 (+ 600us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.661756 (+ 717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.661763 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.662141 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:05.662428 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.662435 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.664750 (+ 2315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.664751 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.666006 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.666009 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.666861 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.666863 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.666998 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.667623 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:05.667634 (+ 11us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:05.669386 (+ 1752us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:05.671500 (+ 2114us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.671509 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.671511 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.671826 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.672094 (+ 268us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.672095 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.672096 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.672236 (+ 140us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.672649 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:05.672663 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:05.672854 (+ 191us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.672975 (+ 121us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.673158 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":159,"thread_start_us":114,"threads_started":1}
I20260923 23:13:05.674440 490 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:13:05.674536 490 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:05.674719 490 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
I20260923 23:13:05.675424 302 ts_manager.cc:194] Registered new tserver with Master: 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857)
I20260923 23:13:05.676366 302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:55211
I20260923 23:13:05.677176 490 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.129@KRBTEST.COM
I20260923 23:13:05.684165 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.676021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.16.122.129:43857)
0923 23:13:05.676166 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.676170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.676323 (+ 153us) server_negotiation.cc:407] Connection header received
0923 23:13:05.676443 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.676446 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.676460 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.676497 (+ 37us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.677040 (+ 543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.678050 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.679169 (+ 1119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.679458 (+ 289us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.679516 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.681615 (+ 2099us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.681622 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.681623 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.681629 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.682991 (+ 1362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.683451 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.683453 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.683454 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.683506 (+ 52us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.683666 (+ 160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.683667 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.683667 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.683789 (+ 122us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.683842 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.684003 (+ 161us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.684097 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":5,"server-negotiator.queue_time_us":118,"thread_start_us":75,"threads_started":1}
I20260923 23:13:05.684195 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.675960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.0.0.1:49674)
0923 23:13:05.676232 (+ 272us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.676269 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.676355 (+ 86us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.676579 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.676582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.676902 (+ 320us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:05.676958 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.676960 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.678122 (+ 1162us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.678123 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.679091 (+ 968us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.679094 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.679524 (+ 430us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.679525 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.679597 (+ 72us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.680054 (+ 457us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:05.680059 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:05.681536 (+ 1477us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:05.683112 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.683128 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.683131 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.683381 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.683557 (+ 176us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.683557 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.683558 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.683617 (+ 59us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.683838 (+ 221us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:05.683844 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:05.683895 (+ 51us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.683977 (+ 82us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.684093 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":244,"thread_start_us":83,"threads_started":1}
I20260923 23:13:05.684697 422 tablet_service.cc:1514] Processing DeleteTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:45 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49674
I20260923 23:13:05.685209 499 tablet_replica.cc:330] stopping tablet replica
I20260923 23:13:05.685318 499 raft_consensus.cc:2229] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:05.685386 499 raft_consensus.cc:2258] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:05.685704 499 ts_tablet_manager.cc:1905] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:05.688024 499 ts_tablet_manager.cc:1918] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:05.688094 499 log.cc:1196] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal/wals/307a2eecfd6c425d8dac4e07f90a343f
I20260923 23:13:05.688301 499 ts_tablet_manager.cc:1939] T 307a2eecfd6c425d8dac4e07f90a343f P 1defe561276f45bab2b8971a7a0f4a1e: Deleting consensus metadata
I20260923 23:13:05.688849 32756 catalog_manager.cc:4841] TS 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857): tablet 307a2eecfd6c425d8dac4e07f90a343f (table test-table [id=854caf1d509543b09682ae1b4f74885c]) successfully deleted
W20260923 23:13:05.749099 495 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:05.749311 495 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:05.749342 495 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:05.750492 495 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:05.750532 495 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:05.750566 495 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:05.751979 495 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:42609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=40629
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.495
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:05.752219 495 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:05.752420 495 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:05.757207 505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:05.757228 504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.757333 495 server_base.cc:1034] running on GCE node
W20260923 23:13:05.757207 507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.757566 495 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:05.757769 495 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:05.758906 495 hybrid_clock.cc:648] HybridClock initialized: now 1790205185758895 us; error 28 us; skew 500 ppm
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:05.761112 495 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:13:05.761524 495 webserver.cc:460] Webserver started at http://127.16.122.130:40629/ using document root <none> and password file <none>
I20260923 23:13:05.761719 495 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:05.761780 495 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:05.763007 495 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:13:05.763652 514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.763790 495 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:05.763868 495 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "4c80df2d0c564de3a84527857229c856"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "77f462cf3d2a837ddb3dd46f1c0df5b1"
server_key_iv: "644ff7eef19a4334c612319c3ebd8fd9"
server_key_version: "encryptionkey@0"
I20260923 23:13:05.764192 495 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:05.772353 495 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:05.774205 495 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:05.774338 495 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:05.774550 495 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:05.774989 521 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:13:05.776342 495 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:13:05.776382 495 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:13:05.776412 495 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:13:05.777007 495 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:13:05.777040 495 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:13:05.777091 521 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Bootstrap starting.
I20260923 23:13:05.778590 521 log.cc:824] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:05.781437 521 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:13:05.781713 521 tablet_bootstrap.cc:492] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Bootstrap complete.
I20260923 23:13:05.781862 521 ts_tablet_manager.cc:1397] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.002s
I20260923 23:13:05.782630 495 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:42609
I20260923 23:13:05.782707 629 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:42609 every 8 connection(s)
I20260923 23:13:05.782948 495 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:05.783507 521 raft_consensus.cc:348] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:13:05.783830 521 raft_consensus.cc:729] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c80df2d0c564de3a84527857229c856, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.783962 521 consensus_queue.cc:260] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } }
I20260923 23:13:05.784204 521 ts_tablet_manager.cc:1428] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Time spent starting tablet: real 0.002s user 0.004s sys 0.001s
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:05.788964 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 495
I20260923 23:13:05.789217 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:38627
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=34761
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:13:05.791462 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.783670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:54459 (local address 127.16.122.190:41679)
0923 23:13:05.783719 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.783721 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.783986 (+ 265us) server_negotiation.cc:407] Connection header received
0923 23:13:05.784209 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.784211 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.784227 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.784276 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.784864 (+ 588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.785362 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.786375 (+ 1013us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.786594 (+ 219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.786648 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.788925 (+ 2277us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.788926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.788927 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.788934 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.790224 (+ 1290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.790580 (+ 356us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.790581 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.790582 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.790633 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.790840 (+ 207us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.790841 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.790842 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.790947 (+ 105us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.791002 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.791277 (+ 275us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.791369 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:13:05.791788 632 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.783650 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:54459)
0923 23:13:05.783854 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.783900 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.784092 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.784329 (+ 237us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.784333 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.784598 (+ 265us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:05.784775 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.784780 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.785459 (+ 679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.785460 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.786297 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.786300 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.786659 (+ 359us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.786660 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.786772 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.787558 (+ 786us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:05.787565 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:05.788841 (+ 1276us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:05.790296 (+ 1455us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.790303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.790305 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.790516 (+ 211us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.790684 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.790685 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.790685 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.790788 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.790998 (+ 210us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:05.791004 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:05.791162 (+ 158us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.791249 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.791399 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":152,"thread_start_us":115,"threads_started":1}
I20260923 23:13:05.792240 630 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:13:05.792326 630 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:05.792485 630 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
I20260923 23:13:05.793018 302 ts_manager.cc:194] Registered new tserver with Master: 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
I20260923 23:13:05.793522 302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:54459
I20260923 23:13:05.794553 630 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.130@KRBTEST.COM
I20260923 23:13:05.800343 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.793486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42516 (local address 127.16.122.130:42609)
0923 23:13:05.793661 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.793666 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.793675 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:05.793708 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.793710 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.793723 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.793784 (+ 61us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.794063 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.795115 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.795755 (+ 640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.796063 (+ 308us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.796118 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.797879 (+ 1761us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.797886 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.797886 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.797892 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.799308 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.799658 (+ 350us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.799661 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.799661 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.799707 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.799865 (+ 158us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.799866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.799866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.800015 (+ 149us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.800078 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.800208 (+ 130us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.800279 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":144,"thread_start_us":81,"threads_started":1}
I20260923 23:13:05.800383 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.793286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.0.0.1:42516)
0923 23:13:05.793345 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.793357 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.793405 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.793795 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.793796 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.793955 (+ 159us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:05.793994 (+ 39us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.793996 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.795185 (+ 1189us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.795186 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.795691 (+ 505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.795692 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.796125 (+ 433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.796126 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.796190 (+ 64us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.796597 (+ 407us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:05.796600 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:05.797802 (+ 1202us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:05.799378 (+ 1576us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.799379 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.799380 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.799593 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.799755 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.799755 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.799755 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.799812 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.800071 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:05.800071 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:05.800117 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.800185 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.800284 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":34}
I20260923 23:13:05.800855 564 tablet_service.cc:1514] Processing DeleteTablet for tablet 307a2eecfd6c425d8dac4e07f90a343f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:12:45 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:42516
I20260923 23:13:05.801523 636 tablet_replica.cc:330] stopping tablet replica
I20260923 23:13:05.801620 636 raft_consensus.cc:2229] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:05.801687 636 raft_consensus.cc:2258] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:05.801961 636 ts_tablet_manager.cc:1905] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:05.804158 636 ts_tablet_manager.cc:1918] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:05.804216 636 log.cc:1196] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/307a2eecfd6c425d8dac4e07f90a343f
I20260923 23:13:05.804389 636 ts_tablet_manager.cc:1939] T 307a2eecfd6c425d8dac4e07f90a343f P 4c80df2d0c564de3a84527857229c856: Deleting consensus metadata
I20260923 23:13:05.804821 32755 catalog_manager.cc:4841] TS 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): tablet 307a2eecfd6c425d8dac4e07f90a343f (table test-table [id=854caf1d509543b09682ae1b4f74885c]) successfully deleted
W20260923 23:13:05.870558 634 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:05.870779 634 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:05.870811 634 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:05.871987 634 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:05.872028 634 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:05.872062 634 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:05.873616 634 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=34761
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.634
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:05.873860 634 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:05.874083 634 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:05.878911 642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:05.878947 644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:05.878911 641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:05.879148 634 server_base.cc:1034] running on GCE node
I20260923 23:13:05.879338 634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:05.879586 634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:05.880724 634 hybrid_clock.cc:648] HybridClock initialized: now 1790205185880714 us; error 30 us; skew 500 ppm
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:05.883245 634 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:13:05.883704 634 webserver.cc:460] Webserver started at http://127.16.122.131:34761/ using document root <none> and password file <none>
I20260923 23:13:05.883913 634 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:05.883973 634 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:05.885303 634 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:05.885907 651 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.886080 634 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:05.886180 634 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "5b853e63947049789b1e0da12677cab2"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "f5d1ae7d65d95b5f740e9e2f8c690990"
server_key_iv: "dd3cab27bc6663b09a778fcde84e3829"
server_key_version: "encryptionkey@0"
I20260923 23:13:05.886523 634 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:05.917536 634 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:05.919787 634 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:05.919930 634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:05.920148 634 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:05.920452 634 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:05.920485 634 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.920513 634 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:05.920531 634 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:05.926113 634 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:38627
I20260923 23:13:05.926189 764 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:38627 every 8 connection(s)
I20260923 23:13:05.926522 634 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:13:05 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205185, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:05.934286 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 634
I20260923 23:13:05.936558 493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.927243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:58623 (local address 127.16.122.190:41679)
0923 23:13:05.927307 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.927309 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.927569 (+ 260us) server_negotiation.cc:407] Connection header received
0923 23:13:05.928048 (+ 479us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.928050 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.928062 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.928158 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.929383 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.930141 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.931425 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.931720 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.931796 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.933909 (+ 2113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.933910 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.933911 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.933917 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.935324 (+ 1407us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.935688 (+ 364us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.935690 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.935690 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.935748 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.935963 (+ 215us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.935963 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.935964 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.936073 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.936128 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.936388 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.936468 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:05.936782 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.927249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58623)
0923 23:13:05.927431 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.927473 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.927942 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.928258 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.928262 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.928991 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:05.929283 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.929291 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.930234 (+ 943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.930235 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.931304 (+ 1069us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.931307 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.931939 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.931939 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.932051 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.932543 (+ 492us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:05.932552 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:05.933827 (+ 1275us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:05.935398 (+ 1571us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.935405 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.935407 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.935621 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.935801 (+ 180us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:05.935802 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:05.935802 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:05.935909 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:05.936126 (+ 217us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:05.936133 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:05.936269 (+ 136us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.936359 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.936510 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260923 23:13:05.937218 765 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41679
I20260923 23:13:05.937309 765 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:05.937486 765 heartbeater.cc:511] Master 127.16.122.190:41679 requested a full tablet report, sending...
I20260923 23:13:05.937855 302 ts_manager.cc:194] Registered new tserver with Master: 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627)
I20260923 23:13:05.938199 302 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:58623
I20260923 23:13:05.940722 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.934665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41374 (local address 127.16.122.190:41679)
0923 23:13:05.934851 (+ 186us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.934853 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.934986 (+ 133us) server_negotiation.cc:407] Connection header received
0923 23:13:05.935114 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.935115 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.935127 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.935181 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.935721 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.936196 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.937139 (+ 943us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.937369 (+ 230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.937425 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.938562 (+ 1137us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.938564 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.938565 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.938571 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.939761 (+ 1190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.940058 (+ 297us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.940059 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.940060 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.940113 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.940270 (+ 157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.940271 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.940272 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.940372 (+ 100us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.940523 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.940558 (+ 35us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.940632 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":166,"thread_start_us":53,"threads_started":1}
I20260923 23:13:05.941331 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:05.943667 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.941617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41388 (local address 127.16.122.190:41679)
0923 23:13:05.941659 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.941661 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.941742 (+ 81us) server_negotiation.cc:407] Connection header received
0923 23:13:05.941789 (+ 47us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.941790 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.941816 (+ 26us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.941870 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:05.941944 (+ 74us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.942379 (+ 435us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.943073 (+ 694us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.943276 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.943325 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.943439 (+ 114us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:05.943478 (+ 39us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0923 23:13:05.943626 (+ 148us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:41388: token expired
Metrics: {"server-negotiator.queue_time_us":25}
W20260923 23:13:05.943706 768 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:41388: token expired
W20260923 23:13:05.943783 32699 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.122.190:41679)
I20260923 23:13:05.948786 768 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.944005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41402 (local address 127.16.122.190:41679)
0923 23:13:05.944042 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.944044 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.944079 (+ 35us) server_negotiation.cc:407] Connection header received
0923 23:13:05.944123 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.944123 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.944130 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.944180 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:05.944550 (+ 370us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.944970 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.945590 (+ 620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.945760 (+ 170us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.945809 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.946825 (+ 1016us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:05.946826 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:05.946826 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:05.946832 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:05.948035 (+ 1203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.948290 (+ 255us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.948291 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.948292 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.948333 (+ 41us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:05.948448 (+ 115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:05.948449 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:05.948449 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:05.948528 (+ 79us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:05.948592 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.948655 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.948710 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:13:05.949905 302 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41402:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:05.951488 302 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:05.953405 422 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df (DEFAULT_TABLE table=test-table [id=2bf831b26b2d4b3998f3c45ed62dba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:05.953585 564 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df (DEFAULT_TABLE table=test-table [id=2bf831b26b2d4b3998f3c45ed62dba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:05.953624 422 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd21f6eabbd4bd8b995230e5e3d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:05.953753 564 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd21f6eabbd4bd8b995230e5e3d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:05.955241 382 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap starting.
I20260923 23:13:05.955310 521 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Bootstrap starting.
I20260923 23:13:05.955495 382 tablet_bootstrap.cc:654] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:05.955703 521 tablet_bootstrap.cc:654] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:05.955971 382 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: No bootstrap required, opened a new log
I20260923 23:13:05.956022 382 ts_tablet_manager.cc:1397] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Time spent bootstrapping tablet: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:05.956359 382 raft_consensus.cc:348] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.956434 382 raft_consensus.cc:374] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:05.956482 382 raft_consensus.cc:729] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defe561276f45bab2b8971a7a0f4a1e, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.956532 382 consensus_queue.cc:260] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.956629 382 ts_tablet_manager.cc:1428] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 23:13:05.957010 521 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: No bootstrap required, opened a new log
I20260923 23:13:05.957062 521 ts_tablet_manager.cc:1397] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.000s
I20260923 23:13:05.957214 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.953137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:38627 (local address 127.0.0.1:33408)
0923 23:13:05.953205 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:05.953214 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:05.953243 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:05.953595 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:05.953596 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:05.953598 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:05.953639 (+ 41us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.953640 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.954756 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.954757 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.955789 (+ 1032us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:05.955793 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.956954 (+ 1161us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.956954 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:05.957025 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.957041 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 23:13:05.957105 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:13:05.957151 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 23:13:05.957267 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:05.953216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33408 (local address 127.16.122.131:38627)
0923 23:13:05.953416 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:05.953422 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:05.953431 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:05.953470 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:05.953472 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:05.953487 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:05.953706 (+ 219us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:05.953790 (+ 84us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.954691 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.956082 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:05.956855 (+ 773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:05.956956 (+ 101us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:05.957034 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:05.957124 (+ 90us) server_negotiation.cc:299] Negotiation successful
0923 23:13:05.957172 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162,"thread_start_us":104,"threads_started":1}
I20260923 23:13:05.957345 521 raft_consensus.cc:348] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.957417 521 raft_consensus.cc:374] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:05.957471 521 raft_consensus.cc:729] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4c80df2d0c564de3a84527857229c856, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.957515 521 consensus_queue.cc:260] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.957612 521 ts_tablet_manager.cc:1428] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20260923 23:13:05.957994 699 tablet_service.cc:1467] Processing CreateTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df (DEFAULT_TABLE table=test-table [id=2bf831b26b2d4b3998f3c45ed62dba4f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:05.958292 699 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8bd21f6eabbd4bd8b995230e5e3d95df. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:05.961117 774 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Bootstrap starting.
I20260923 23:13:05.961717 774 tablet_bootstrap.cc:654] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:05.962136 774 log.cc:824] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:05.962728 774 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: No bootstrap required, opened a new log
I20260923 23:13:05.962778 774 ts_tablet_manager.cc:1397] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:05.964097 774 raft_consensus.cc:348] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.964241 774 raft_consensus.cc:374] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:05.964263 774 raft_consensus.cc:729] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b853e63947049789b1e0da12677cab2, State: Initialized, Role: FOLLOWER
I20260923 23:13:05.964356 774 consensus_queue.cc:260] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:05.964586 774 ts_tablet_manager.cc:1428] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:05.964661 765 heartbeater.cc:503] Master 127.16.122.190:41679 was elected leader, sending a full tablet report...
I20260923 23:13:06.015820 776 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:06.015942 776 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:06.016254 776 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609), 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857)
I20260923 23:13:06.019523 498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.016517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:40803 (local address 127.16.122.129:43857)
0923 23:13:06.016571 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:06.016573 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:06.016582 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:06.016615 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:06.016616 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:06.016643 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:06.016721 (+ 78us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:06.016875 (+ 154us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.017452 (+ 577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.018743 (+ 1291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.019199 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.019265 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:06.019343 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:06.019414 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 23:13:06.019452 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:13:06.019523 767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.016438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.16.122.131:40803)
0923 23:13:06.016490 (+ 52us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:06.016517 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:06.016561 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:06.016734 (+ 173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:06.016734 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:06.016736 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:06.016811 (+ 75us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:06.016813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.017525 (+ 712us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.017526 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:06.018647 (+ 1121us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:06.018650 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.019269 (+ 619us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.019270 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:06.019349 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:06.019353 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:06.019407 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:13:06.019447 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:13:06.019814 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd21f6eabbd4bd8b995230e5e3d95df" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defe561276f45bab2b8971a7a0f4a1e" is_pre_election: true
I20260923 23:13:06.019965 435 raft_consensus.cc:2454] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 0.
I20260923 23:13:06.020161 777 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.016447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.16.122.131:51289)
0923 23:13:06.016650 (+ 203us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:06.016660 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:06.016717 (+ 57us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:06.017534 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:06.017535 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:06.017537 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:06.017602 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:06.017604 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.018507 (+ 903us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.018508 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:06.019278 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:06.019280 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.019928 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.019929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:06.019997 (+ 68us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:06.020002 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:13:06.020062 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:13:06.020098 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":184,"mutex_wait_us":47,"thread_start_us":95,"threads_started":1}
I20260923 23:13:06.020198 654 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e, 5b853e63947049789b1e0da12677cab2; no voters:
I20260923 23:13:06.020265 635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.016510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:51289 (local address 127.16.122.130:42609)
0923 23:13:06.016558 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:06.016560 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:06.016724 (+ 164us) server_negotiation.cc:407] Connection header received
0923 23:13:06.016761 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:06.016762 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:06.016789 (+ 27us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:06.016869 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:06.017686 (+ 817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.018438 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.019356 (+ 918us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.019852 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.019926 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:06.019997 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:06.020139 (+ 142us) server_negotiation.cc:299] Negotiation successful
0923 23:13:06.020186 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:13:06.020363 776 raft_consensus.cc:2781] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:06.020439 776 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:06.020469 776 raft_consensus.cc:3037] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:06.020583 584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd21f6eabbd4bd8b995230e5e3d95df" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c80df2d0c564de3a84527857229c856" is_pre_election: true
I20260923 23:13:06.020707 584 raft_consensus.cc:2454] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 0.
I20260923 23:13:06.021234 776 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:06.021359 776 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 election: Requested vote from peers 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609), 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857)
I20260923 23:13:06.021488 584 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd21f6eabbd4bd8b995230e5e3d95df" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4c80df2d0c564de3a84527857229c856"
I20260923 23:13:06.021504 435 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8bd21f6eabbd4bd8b995230e5e3d95df" candidate_uuid: "5b853e63947049789b1e0da12677cab2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1defe561276f45bab2b8971a7a0f4a1e"
I20260923 23:13:06.021574 435 raft_consensus.cc:3037] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:06.021574 584 raft_consensus.cc:3037] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:06.022414 584 raft_consensus.cc:2454] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 1.
I20260923 23:13:06.022430 435 raft_consensus.cc:2454] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b853e63947049789b1e0da12677cab2 in term 1.
I20260923 23:13:06.022578 654 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e, 5b853e63947049789b1e0da12677cab2; no voters:
I20260923 23:13:06.022678 776 raft_consensus.cc:2781] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:06.022817 776 raft_consensus.cc:686] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 1 LEADER]: Becoming Leader. State: Replica: 5b853e63947049789b1e0da12677cab2, State: Running, Role: LEADER
I20260923 23:13:06.022905 776 consensus_queue.cc:237] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:06.023772 302 catalog_manager.cc:5495] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b853e63947049789b1e0da12677cab2 (127.16.122.131). New cstate: current_term: 1 leader_uuid: "5b853e63947049789b1e0da12677cab2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } health_report { overall_health: UNKNOWN } } }
I20260923 23:13:06.029253 771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.026892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33422 (local address 127.16.122.131:38627)
0923 23:13:06.026942 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:06.026943 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:06.026951 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:06.026994 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:06.026994 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:06.027008 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:06.027099 (+ 91us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:06.027184 (+ 85us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.027624 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.028451 (+ 827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:06.028723 (+ 272us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:06.028790 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:06.028926 (+ 136us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:06.029050 (+ 124us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:06.029131 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:06.029148 (+ 17us) server_negotiation.cc:299] Negotiation successful
0923 23:13:06.029169 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:06.031523 435 raft_consensus.cc:1261] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Refusing update from remote peer 5b853e63947049789b1e0da12677cab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:06.031565 584 raft_consensus.cc:1261] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Refusing update from remote peer 5b853e63947049789b1e0da12677cab2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:06.031833 778 consensus_queue.cc:1035] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:06.031949 776 consensus_queue.cc:1035] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260923 23:13:06.033404 631 tablet.cc:2369] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:13:06.033937 780 mvcc.cc:205] Tried to move back new op lower bound from 7332680441982668800 to 7332680441950621696. Current Snapshot: MvccSnapshot[applied={T|T < 7332680441982668800}]
I20260923 23:13:06.034722 781 mvcc.cc:205] Tried to move back new op lower bound from 7332680441982668800 to 7332680441950621696. Current Snapshot: MvccSnapshot[applied={T|T < 7332680441982668800}]
I20260923 23:13:06.035929 302 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:06.036023 302 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41402:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:06.036675 302 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Sending DeleteTablet for 3 replicas of tablet 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:06.036948 422 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:06 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49674
I20260923 23:13:06.036940 699 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:06 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:33408
I20260923 23:13:06.036955 564 tablet_service.cc:1514] Processing DeleteTablet for tablet 8bd21f6eabbd4bd8b995230e5e3d95df with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:06 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:42516
I20260923 23:13:06.037032 636 tablet_replica.cc:330] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: stopping tablet replica
I20260923 23:13:06.037035 499 tablet_replica.cc:330] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: stopping tablet replica
I20260923 23:13:06.037073 636 raft_consensus.cc:2229] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:06.037079 499 raft_consensus.cc:2229] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:06.037109 636 raft_consensus.cc:2258] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:06.037114 499 raft_consensus.cc:2258] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:06.037317 790 tablet_replica.cc:330] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: stopping tablet replica
I20260923 23:13:06.037393 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 340
I20260923 23:13:06.037483 790 raft_consensus.cc:2229] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:06.037482 499 ts_tablet_manager.cc:1905] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:06.037644 790 raft_consensus.cc:2258] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:06.038064 790 ts_tablet_manager.cc:1905] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:06.039572 790 ts_tablet_manager.cc:1918] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:06.039634 790 log.cc:1196] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:06.039876 790 ts_tablet_manager.cc:1939] T 8bd21f6eabbd4bd8b995230e5e3d95df P 5b853e63947049789b1e0da12677cab2: Deleting consensus metadata
I20260923 23:13:06.040118 636 ts_tablet_manager.cc:1905] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:06.041133 32755 catalog_manager.cc:4841] TS 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): tablet 8bd21f6eabbd4bd8b995230e5e3d95df (table test-table [id=2bf831b26b2d4b3998f3c45ed62dba4f]) successfully deleted
I20260923 23:13:06.041270 636 ts_tablet_manager.cc:1918] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:06.041320 636 log.cc:1196] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:06.041491 636 ts_tablet_manager.cc:1939] T 8bd21f6eabbd4bd8b995230e5e3d95df P 4c80df2d0c564de3a84527857229c856: Deleting consensus metadata
I20260923 23:13:06.041765 32755 catalog_manager.cc:4841] TS 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): tablet 8bd21f6eabbd4bd8b995230e5e3d95df (table test-table [id=2bf831b26b2d4b3998f3c45ed62dba4f]) successfully deleted
W20260923 23:13:06.044271 32756 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:43857 (error 108)
I20260923 23:13:06.044550 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 495
I20260923 23:13:06.044714 497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:06.044602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43857 (local address 127.0.0.1:49688)
0923 23:13:06.044648 (+ 46us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:06.044685 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43857: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:13:06.045250 32756 catalog_manager.cc:4584] TS 1defe561276f45bab2b8971a7a0f4a1e (127.16.122.129:43857): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 8bd21f6eabbd4bd8b995230e5e3d95df: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43857: connect: Connection refused (error 111)
I20260923 23:13:06.048964 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 634
I20260923 23:13:06.053004 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 32737
I20260923 23:13:26.057886 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--webserver_interface=127.16.122.190
--webserver_port=43563
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41679
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:26.141855 792 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:26.142086 792 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:26.142119 792 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:26.143368 792 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260923 23:13:26.143421 792 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260923 23:13:26.143433 792 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260923 23:13:26.143443 792 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:26.143455 792 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:26.143466 792 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:26.143477 792 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:26.143487 792 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:26.144953 792 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41679
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=43563
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.792
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:26.145157 792 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:26.145356 792 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:26.150004 797 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.150010 800 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.149987 798 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:26.150189 792 server_base.cc:1034] running on GCE node
I20260923 23:13:26.150545 792 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:26.150823 792 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:26.151968 792 hybrid_clock.cc:648] HybridClock initialized: now 1790205206151957 us; error 27 us; skew 500 ppm
Sep 23 23:13:26 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:26.158356 792 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:13:26.158838 792 webserver.cc:460] Webserver started at http://127.16.122.190:43563/ using document root <none> and password file <none>
I20260923 23:13:26.159107 792 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:26.159173 792 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:26.160542 792 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:26.161154 807 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.161305 792 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:26.161397 792 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "1157dc839a93cbf938e9e9e0ae74bfe5"
server_key_iv: "c51effb824876359d594021e4765657e"
server_key_version: "encryptionkey@0"
I20260923 23:13:26.161734 792 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:26.173192 792 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:26.175086 792 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:26.175241 792 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:26.178951 792 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41679
I20260923 23:13:26.178994 859 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41679 every 8 connection(s)
I20260923 23:13:26.179399 792 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:26.180874 860 sys_catalog.cc:263] Verifying existing consensus state
I20260923 23:13:26.181502 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap starting.
I20260923 23:13:26.182794 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 792
I20260923 23:13:26.183117 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:43857
--local_ip_for_outbound_sockets=127.16.122.129
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=39313
--webserver_interface=127.16.122.129
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:13:26.190393 860 log.cc:824] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:26.192757 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:13:26.193029 860 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Bootstrap complete.
I20260923 23:13:26.195183 860 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:26.195433 860 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Initialized, Role: FOLLOWER
I20260923 23:13:26.195596 860 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:26.195654 860 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:26.195693 860 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:26.195736 860 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 2 FOLLOWER]: Advancing to term 3
I20260923 23:13:26.196761 860 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:26.196874 860 leader_election.cc:302] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6803b37ab8ef4b0da1e3b61b9ebb3f23; no voters:
I20260923 23:13:26.197062 860 leader_election.cc:288] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [CANDIDATE]: Term 3 election: Requested vote from peers
I20260923 23:13:26.197113 864 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 3 FOLLOWER]: Leader election won for term 3
I20260923 23:13:26.197230 864 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [term 3 LEADER]: Becoming Leader. State: Replica: 6803b37ab8ef4b0da1e3b61b9ebb3f23, State: Running, Role: LEADER
I20260923 23:13:26.197314 864 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } }
I20260923 23:13:26.197345 860 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:26.197723 866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:13:26.197935 866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:26.197728 864 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6803b37ab8ef4b0da1e3b61b9ebb3f23. Latest consensus state: current_term: 3 leader_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6803b37ab8ef4b0da1e3b61b9ebb3f23" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41679 } } }
I20260923 23:13:26.198186 864 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:26.198401 873 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:26.199247 873 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:26.199353 873 catalog_manager.cc:1259] Loaded cluster ID: a366cf80f260477186b90ec8e6702566
I20260923 23:13:26.199383 873 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:26.200477 873 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:26.200667 873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Loaded TSK: 0
I20260923 23:13:26.200716 873 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 6803b37ab8ef4b0da1e3b61b9ebb3f23: Loaded TSK: 1
F20260923 23:13:26.200960 873 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f63241759ad google::LogMessage::Fail() at ??:0
@ 0x7f632417888a google::LogMessage::SendToLog() at ??:0
@ 0x7f63241754de google::LogMessage::Flush() at ??:0
@ 0x7f6324176a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f63268e2104 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7f63268dfd41 kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7f632c0cf1cd kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7f632c0cf48e kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7f632c0cf594 _ZNSt17_Function_handlerIFN4kudu6StatusEvEZNS0_6master14CatalogManager24PrepareForLeadershipTaskEvEUlvE3_E9_M_invokeERKSt9_Any_data at ??:0
@ 0x7f632c095c48 _ZZN4kudu6master14CatalogManager24PrepareForLeadershipTaskEvENKUlSt8functionIFNS_6StatusEvEERKNS_9consensus13RaftConsensusElPKcE_clES5_S9_lSB_.isra.591 at ??:0
@ 0x7f632c09fad0 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f63260430d3 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f6326038954 kudu::Thread::SuperviseThread() at ??:0
@ 0x7f6326e9c6db start_thread at ??:0
@ 0x7f6322c7361f clone at ??:0
W20260923 23:13:26.273406 862 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:26.273635 862 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:26.273671 862 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:26.274845 862 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:26.274883 862 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:26.274919 862 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:26.276432 862 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:43857
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=39313
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.862
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:26.276670 862 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:26.276878 862 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:26.283105 901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.283145 902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.283281 904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:26.283341 862 server_base.cc:1034] running on GCE node
I20260923 23:13:26.283571 862 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:26.283824 862 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:26.284991 862 hybrid_clock.cc:648] HybridClock initialized: now 1790205206284976 us; error 25 us; skew 500 ppm
Sep 23 23:13:26 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:26.288103 862 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:13:26.288637 862 webserver.cc:460] Webserver started at http://127.16.122.129:39313/ using document root <none> and password file <none>
I20260923 23:13:26.288901 862 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:26.288961 862 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:26.290766 862 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:26.291646 911 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.291810 862 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:26.291909 862 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "1defe561276f45bab2b8971a7a0f4a1e"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "26bc18b0898d29c3e5d39bd82f238ac8"
server_key_iv: "d9c3bdef2893b1f30ed574fce2247387"
server_key_version: "encryptionkey@0"
I20260923 23:13:26.292369 862 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:26.331748 862 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:26.334436 862 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:26.334601 862 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:26.334906 862 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:26.335455 918 ts_tablet_manager.cc:536] Loading tablet metadata (0/1 complete)
I20260923 23:13:26.336896 862 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260923 23:13:26.336959 862 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.002s user 0.000s sys 0.000s
I20260923 23:13:26.336995 862 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260923 23:13:26.337904 862 ts_tablet_manager.cc:610] Registered 1 tablets
I20260923 23:13:26.337954 862 ts_tablet_manager.cc:589] Time spent register tablets: real 0.001s user 0.000s sys 0.000s
I20260923 23:13:26.338033 918 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap starting.
I20260923 23:13:26.339728 918 log.cc:824] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:26.342674 918 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260923 23:13:26.342988 918 tablet_bootstrap.cc:492] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Bootstrap complete.
I20260923 23:13:26.343178 918 ts_tablet_manager.cc:1397] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260923 23:13:26.345054 862 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:43857
I20260923 23:13:26.344856 918 raft_consensus.cc:348] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:26.345212 918 raft_consensus.cc:729] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1defe561276f45bab2b8971a7a0f4a1e, State: Initialized, Role: FOLLOWER
I20260923 23:13:26.345175 1026 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:43857 every 8 connection(s)
I20260923 23:13:26.345458 862 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:26.345422 918 consensus_queue.cc:260] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:26.345762 918 ts_tablet_manager.cc:1428] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:26.348912 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 862
I20260923 23:13:26.349205 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:42609
--local_ip_for_outbound_sockets=127.16.122.130
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=40629
--webserver_interface=127.16.122.130
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:26.408727 1030 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:13:26.346579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:58737)
0923 23:13:26.346769 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.346805 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:26.346932 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:26.408508 (+ 61576us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"client-negotiator.queue_time_us":145,"thread_start_us":89,"threads_started":1}
W20260923 23:13:26.408950 1028 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:41679 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:41679: Client connection negotiation failed: client connection to 127.16.122.190:41679: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260923 23:13:26.409286 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.409166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:59245)
0923 23:13:26.409211 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.409254 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:13:26.409615 1030 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.409548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:42353)
0923 23:13:26.409581 (+ 33us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.409601 (+ 20us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
W20260923 23:13:26.409732 1028 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:13:26.433451 1031 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:26.433729 1031 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:26.433763 1031 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:26.435006 1031 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:26.435094 1031 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:26.435133 1031 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:26.436621 1031 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:42609
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=40629
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1031
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:26.436849 1031 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:26.437064 1031 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:26.442363 1037 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.442370 1039 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:26.442400 1031 server_base.cc:1034] running on GCE node
W20260923 23:13:26.442389 1036 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:26.442775 1031 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:26.443027 1031 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:26.444195 1031 hybrid_clock.cc:648] HybridClock initialized: now 1790205206444185 us; error 27 us; skew 500 ppm
Sep 23 23:13:26 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:26.446542 1031 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:13:26.446964 1031 webserver.cc:460] Webserver started at http://127.16.122.130:40629/ using document root <none> and password file <none>
I20260923 23:13:26.447196 1031 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:26.447256 1031 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:26.448565 1031 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:26.449093 1046 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.449275 1031 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:26.449355 1031 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "4c80df2d0c564de3a84527857229c856"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "77f462cf3d2a837ddb3dd46f1c0df5b1"
server_key_iv: "644ff7eef19a4334c612319c3ebd8fd9"
server_key_version: "encryptionkey@0"
I20260923 23:13:26.449697 1031 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:26.464301 1031 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:26.466359 1031 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:26.466506 1031 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:26.466868 1031 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:26.467223 1031 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:26.467270 1031 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.467300 1031 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:26.467324 1031 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.473255 1031 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:42609
I20260923 23:13:26.473310 1159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:42609 every 8 connection(s)
I20260923 23:13:26.473641 1031 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:26.474498 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1031
I20260923 23:13:26.474661 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.474259 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:38001)
0923 23:13:26.474419 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.474491 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122,"thread_start_us":87,"threads_started":1}
I20260923 23:13:26.474759 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:38627
--local_ip_for_outbound_sockets=127.16.122.131
--tserver_master_addrs=127.16.122.190:41679
--webserver_port=34761
--webserver_interface=127.16.122.131
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33957
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:26.475086 1160 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:41679 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:41679: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
I20260923 23:13:26.475359 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.475268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:44229)
0923 23:13:26.475309 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.475335 (+ 26us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:13:26.475679 1162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.475613 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:42945)
0923 23:13:26.475647 (+ 34us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.475666 (+ 19us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 23:13:26.475806 1160 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260923 23:13:26.554693 1163 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:26.554916 1163 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:26.554940 1163 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:26.556095 1163 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:26.556139 1163 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:26.556165 1163 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:26.557583 1163 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33957
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:38627
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=34761
--tserver_master_addrs=127.16.122.190:41679
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1163
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:26.557789 1163 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:26.557978 1163 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:26.562973 1171 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.563077 1169 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:26.562973 1168 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:26.563162 1163 server_base.cc:1034] running on GCE node
I20260923 23:13:26.563385 1163 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:26.563658 1163 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:26.564813 1163 hybrid_clock.cc:648] HybridClock initialized: now 1790205206564797 us; error 32 us; skew 500 ppm
Sep 23 23:13:26 dist-test-slave-95hh krb5kdc[32190](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:26.567346 1163 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:13:26.567780 1163 webserver.cc:460] Webserver started at http://127.16.122.131:34761/ using document root <none> and password file <none>
I20260923 23:13:26.567958 1163 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:26.568007 1163 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:26.569322 1163 fs_manager.cc:709] Time spent opening directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:26.569921 1178 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.570070 1163 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.570154 1163 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "5b853e63947049789b1e0da12677cab2"
format_stamp: "Formatted at 2026-09-23 23:12:44 on dist-test-slave-95hh"
server_key: "f5d1ae7d65d95b5f740e9e2f8c690990"
server_key_iv: "dd3cab27bc6663b09a778fcde84e3829"
server_key_version: "encryptionkey@0"
I20260923 23:13:26.570521 1163 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:26.601577 1163 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:26.603560 1163 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:26.603711 1163 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:26.603937 1163 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:26.604251 1163 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:26.604287 1163 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.604316 1163 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:26.604334 1163 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:26.609802 1163 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:38627
I20260923 23:13:26.609864 1291 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:38627 every 8 connection(s)
I20260923 23:13:26.610193 1163 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:13:26.611307 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.610878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:48485)
0923 23:13:26.611074 (+ 196us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.611159 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":69,"threads_started":1}
W20260923 23:13:26.611675 1292 heartbeater.cc:644] Failed to heartbeat to 127.16.122.190:41679 (0 consecutive failures): Network error: Failed to ping master at 127.16.122.190:41679: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
I20260923 23:13:26.611934 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.611843 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:55847)
0923 23:13:26.611887 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.611915 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:13:26.612143 1294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:26.612081 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:44603)
0923 23:13:26.612113 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:26.612131 (+ 18us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":22}
W20260923 23:13:26.612224 1292 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260923 23:13:26.620054 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1163
I20260923 23:13:27.410349 1296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.410096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:39777)
0923 23:13:27.410239 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:27.410279 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":66,"threads_started":1}
I20260923 23:13:27.476326 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.476104 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:40135)
0923 23:13:27.476234 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:27.476273 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112,"thread_start_us":53,"threads_started":1}
I20260923 23:13:27.612800 1298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.612562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:56479)
0923 23:13:27.612706 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:27.612747 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":72,"threads_started":1}
I20260923 23:13:27.914368 1299 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:27.914486 1299 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:27.914808 1299 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
Sep 23 23:13:27 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.131@KRBTEST.COM
Sep 23 23:13:27 dist-test-slave-95hh krb5kdc[32190](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205206, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.130@KRBTEST.COM
I20260923 23:13:27.924479 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.915098 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:43859 (local address 127.16.122.131:38627)
0923 23:13:27.915277 (+ 179us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:27.915289 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:27.915325 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:13:27.915779 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:27.915782 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:27.915824 (+ 42us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:27.915957 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:27.916708 (+ 751us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.917621 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.918602 (+ 981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.918992 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.919115 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:27.921223 (+ 2108us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:27.921227 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:27.921228 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:27.921238 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:27.922904 (+ 1666us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923367 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:27.923370 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:27.923370 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:27.923458 (+ 88us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923678 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:27.923679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:27.923679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:27.923911 (+ 232us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:27.923982 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:27.924242 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 23:13:27.924367 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":145,"thread_start_us":69,"threads_started":1}
I20260923 23:13:27.924489 1303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.915256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:48391 (local address 127.16.122.130:42609)
0923 23:13:27.915390 (+ 134us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:27.915397 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:27.915430 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:13:27.915828 (+ 398us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:27.915830 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:27.915868 (+ 38us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:27.915985 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:27.916885 (+ 900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.917719 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.918602 (+ 883us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.918989 (+ 387us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.919116 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:27.921613 (+ 2497us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:27.921617 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:27.921617 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:27.921627 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:27.922972 (+ 1345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923359 (+ 387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:27.923362 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:27.923363 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:27.923445 (+ 82us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923678 (+ 233us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:27.923679 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:27.923679 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:27.923905 (+ 226us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:27.923970 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:27.924238 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 23:13:27.924384 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":49,"threads_started":1}
I20260923 23:13:27.924500 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.914975 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:38627 (local address 127.16.122.129:43859)
0923 23:13:27.915105 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:27.915115 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:27.915188 (+ 73us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:27.916013 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:27.916016 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:27.916380 (+ 364us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:27.916584 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:27.916589 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.917712 (+ 1123us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.917713 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:27.918523 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:27.918525 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.919088 (+ 563us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.919088 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:27.919209 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:27.919735 (+ 526us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:27.919743 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:27.921123 (+ 1380us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:27.922984 (+ 1861us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:27.922991 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:27.922993 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:27.923288 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:27.923520 (+ 232us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923521 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:27.923521 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:27.923620 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:27.923978 (+ 358us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:27.923979 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:27.924094 (+ 115us) client_negotiation.cc:769] Sending connection context
0923 23:13:27.924181 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:13:27.924364 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":114,"mutex_wait_us":4,"thread_start_us":41,"threads_started":1}
I20260923 23:13:27.925007 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:27.915143 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:42609 (local address 127.16.122.129:48391)
0923 23:13:27.915260 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:27.915268 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:27.915303 (+ 35us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:27.916015 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:27.916017 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:27.916386 (+ 369us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:27.916789 (+ 403us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:27.916792 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.917807 (+ 1015us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.917808 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:27.918521 (+ 713us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:27.918523 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:27.919085 (+ 562us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:27.919086 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:27.919210 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:27.919772 (+ 562us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:27.919776 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:27.921533 (+ 1757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:27.923066 (+ 1533us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923067 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:27.923068 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:27.923282 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:27.923505 (+ 223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:27.923506 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:27.923506 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:27.923621 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:27.923968 (+ 347us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:27.923976 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:27.924094 (+ 118us) client_negotiation.cc:769] Sending connection context
0923 23:13:27.924181 (+ 87us) client_negotiation.cc:240] Negotiation successful
0923 23:13:27.924436 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":106,"mutex_wait_us":151,"thread_start_us":36,"threads_started":1}
W20260923 23:13:27.925621 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
W20260923 23:13:27.925812 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:27.925874 913 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e; no voters: 4c80df2d0c564de3a84527857229c856, 5b853e63947049789b1e0da12677cab2
I20260923 23:13:27.925988 1299 raft_consensus.cc:2726] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:13:28.411022 1302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:28.410846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:46739)
0923 23:13:28.410901 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:28.410950 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:13:28.476958 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:28.476755 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:34143)
0923 23:13:28.476872 (+ 117us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:28.476916 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97,"thread_start_us":60,"threads_started":1}
I20260923 23:13:28.613433 1305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:28.613240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58031)
0923 23:13:28.613350 (+ 110us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:28.613394 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93,"thread_start_us":55,"threads_started":1}
I20260923 23:13:29.411787 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:29.411573 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:40175)
0923 23:13:29.411692 (+ 119us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:29.411738 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":65,"threads_started":1}
I20260923 23:13:29.477634 1307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:29.477379 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:43685)
0923 23:13:29.477543 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:29.477588 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":89,"threads_started":1}
I20260923 23:13:29.614194 1308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:29.613898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58585)
0923 23:13:29.614080 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:29.614139 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160,"thread_start_us":96,"threads_started":1}
I20260923 23:13:29.829294 1309 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:29.829399 1309 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:29.829543 1309 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
W20260923 23:13:29.829986 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
W20260923 23:13:29.830066 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:29.830098 913 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e; no voters: 4c80df2d0c564de3a84527857229c856, 5b853e63947049789b1e0da12677cab2
I20260923 23:13:29.830194 1309 raft_consensus.cc:2726] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:13:30.412551 1310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:30.412332 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:48959)
0923 23:13:30.412452 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:30.412501 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102,"thread_start_us":64,"threads_started":1}
I20260923 23:13:30.478297 1311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:30.478087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:46441)
0923 23:13:30.478217 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:30.478254 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":73,"threads_started":1}
I20260923 23:13:30.614812 1312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:30.614620 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:52269)
0923 23:13:30.614728 (+ 108us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:30.614765 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":60,"threads_started":1}
I20260923 23:13:31.413290 1313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:31.413061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:35463)
0923 23:13:31.413204 (+ 143us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:31.413242 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":84,"threads_started":1}
I20260923 23:13:31.478900 1314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:31.478695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:54491)
0923 23:13:31.478825 (+ 130us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:31.478858 (+ 33us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":78,"threads_started":1}
I20260923 23:13:31.615533 1315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:31.615306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:52541)
0923 23:13:31.615447 (+ 141us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:31.615488 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":74,"threads_started":1}
I20260923 23:13:32.063460 1316 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:32.063557 1316 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:32.063683 1316 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
W20260923 23:13:32.064080 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
W20260923 23:13:32.064174 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:32.064206 913 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e; no voters: 4c80df2d0c564de3a84527857229c856, 5b853e63947049789b1e0da12677cab2
I20260923 23:13:32.064272 1316 raft_consensus.cc:2726] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:13:32.414002 1317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:32.413753 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:60269)
0923 23:13:32.413906 (+ 153us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:32.413951 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":63,"threads_started":1}
I20260923 23:13:32.479560 1318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:32.479324 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:51275)
0923 23:13:32.479460 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:32.479502 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":63,"threads_started":1}
I20260923 23:13:32.616287 1319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:32.616048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58163)
0923 23:13:32.616195 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:32.616241 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126,"thread_start_us":64,"threads_started":1}
I20260923 23:13:33.414705 1320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:33.414487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:46817)
0923 23:13:33.414612 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:33.414656 (+ 44us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":64,"threads_started":1}
I20260923 23:13:33.480230 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:33.480018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:57601)
0923 23:13:33.480140 (+ 122us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:33.480183 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104,"thread_start_us":62,"threads_started":1}
I20260923 23:13:33.616964 1322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:33.616716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58633)
0923 23:13:33.616871 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:33.616912 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130,"thread_start_us":70,"threads_started":1}
I20260923 23:13:34.415311 1323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:34.415083 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:43745)
0923 23:13:34.415218 (+ 135us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:34.415261 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117,"thread_start_us":59,"threads_started":1}
I20260923 23:13:34.480926 1324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:34.480688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:33247)
0923 23:13:34.480836 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:34.480879 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131,"thread_start_us":79,"threads_started":1}
I20260923 23:13:34.617647 1325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:34.617423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:44485)
0923 23:13:34.617557 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:34.617600 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":64,"threads_started":1}
I20260923 23:13:34.899545 1326 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:34.899647 1326 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:34.899816 1326 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
W20260923 23:13:34.900197 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
W20260923 23:13:34.900276 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:34.900306 913 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e; no voters: 4c80df2d0c564de3a84527857229c856, 5b853e63947049789b1e0da12677cab2
I20260923 23:13:34.900386 1326 raft_consensus.cc:2726] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:13:35.415957 1327 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:35.415733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:46579)
0923 23:13:35.415869 (+ 136us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:35.415908 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":64,"threads_started":1}
I20260923 23:13:35.481549 1328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:35.481329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:39167)
0923 23:13:35.481467 (+ 138us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:35.481506 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119,"thread_start_us":60,"threads_started":1}
I20260923 23:13:35.618373 1329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:35.618158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:35135)
0923 23:13:35.618272 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:35.618313 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98,"thread_start_us":61,"threads_started":1}
I20260923 23:13:36.416713 1330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:36.416479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:41067)
0923 23:13:36.416624 (+ 145us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:36.416666 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":128,"thread_start_us":74,"threads_started":1}
I20260923 23:13:36.482286 1331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:36.482046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:54555)
0923 23:13:36.482194 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:36.482239 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":79,"threads_started":1}
I20260923 23:13:36.619149 1332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:36.618874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:52633)
0923 23:13:36.619055 (+ 181us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:36.619102 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137,"thread_start_us":79,"threads_started":1}
I20260923 23:13:36.827324 1333 raft_consensus.cc:482] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:36.827419 1333 raft_consensus.cc:504] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5b853e63947049789b1e0da12677cab2" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 38627 } } peers { permanent_uuid: "4c80df2d0c564de3a84527857229c856" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 42609 } } peers { permanent_uuid: "1defe561276f45bab2b8971a7a0f4a1e" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43857 } }
I20260923 23:13:36.827555 1333 leader_election.cc:288] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627), 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609)
W20260923 23:13:36.827936 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 4c80df2d0c564de3a84527857229c856 (127.16.122.130:42609): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
W20260923 23:13:36.828011 913 leader_election.cc:341] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 5b853e63947049789b1e0da12677cab2 (127.16.122.131:38627): Not found: Tablet not found: 8bd21f6eabbd4bd8b995230e5e3d95df
I20260923 23:13:36.828033 913 leader_election.cc:302] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1defe561276f45bab2b8971a7a0f4a1e; no voters: 4c80df2d0c564de3a84527857229c856, 5b853e63947049789b1e0da12677cab2
I20260923 23:13:36.828119 1333 raft_consensus.cc:2726] T 8bd21f6eabbd4bd8b995230e5e3d95df P 1defe561276f45bab2b8971a7a0f4a1e [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260923 23:13:37.417435 1334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:37.417219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:44999)
0923 23:13:37.417339 (+ 120us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:37.417387 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":63,"threads_started":1}
I20260923 23:13:37.483019 1335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:37.482784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:49949)
0923 23:13:37.482931 (+ 147us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:37.482973 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127,"thread_start_us":79,"threads_started":1}
I20260923 23:13:37.619870 1336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:37.619653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:45057)
0923 23:13:37.619781 (+ 128us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:37.619824 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260923 23:13:38.418151 1337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:38.417926 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:53927)
0923 23:13:38.418052 (+ 126us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:38.418101 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106,"thread_start_us":65,"threads_started":1}
I20260923 23:13:38.483677 1338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:38.483489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:55975)
0923 23:13:38.483603 (+ 114us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:38.483633 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101,"thread_start_us":69,"threads_started":1}
I20260923 23:13:38.620486 1339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:38.620284 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:32881)
0923 23:13:38.620408 (+ 124us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:38.620440 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":108,"thread_start_us":59,"threads_started":1}
I20260923 23:13:39.418787 1340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:39.418570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:59203)
0923 23:13:39.418710 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:39.418742 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123,"thread_start_us":62,"threads_started":1}
I20260923 23:13:39.484337 1341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:39.484106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:45767)
0923 23:13:39.484248 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:39.484289 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":64,"threads_started":1}
I20260923 23:13:39.621261 1342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:39.621016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:57167)
0923 23:13:39.621164 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:39.621213 (+ 49us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":87,"threads_started":1}
I20260923 23:13:40.419435 1343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:40.419218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:60991)
0923 23:13:40.419347 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:40.419385 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":61,"threads_started":1}
I20260923 23:13:40.485055 1344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:40.484775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:51745)
0923 23:13:40.484947 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:40.484995 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":76,"threads_started":1}
I20260923 23:13:40.621990 1345 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:40.621767 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:58697)
0923 23:13:40.621896 (+ 129us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:40.621943 (+ 47us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110,"thread_start_us":64,"threads_started":1}
I20260923 23:13:41.420117 1346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:41.419923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.129:45487)
0923 23:13:41.420032 (+ 109us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:41.420071 (+ 39us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92,"thread_start_us":59,"threads_started":1}
I20260923 23:13:41.485770 1347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:41.485538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.130:39143)
0923 23:13:41.485680 (+ 142us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:41.485716 (+ 36us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124,"thread_start_us":56,"threads_started":1}
I20260923 23:13:41.621253 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 862
I20260923 23:13:41.622778 1348 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:41.622590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41679 (local address 127.16.122.131:33547)
0923 23:13:41.622695 (+ 105us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:41.622733 (+ 38us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.190:41679: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86,"thread_start_us":56,"threads_started":1}
I20260923 23:13:41.625520 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1031
I20260923 23:13:41.629269 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1163
2026-09-23T23:13:41Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (58453 ms)
[----------] 30 tests from SecurityITest (222181 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:41 dist-test-slave-95hh krb5kdc[1351](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:41 dist-test-slave-95hh krb5kdc[1351](info): set up 2 sockets
Sep 23 23:13:41 dist-test-slave-95hh krb5kdc[1351](info): commencing operation
krb5kdc: starting...
W20260923 23:13:42.578475 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:42 dist-test-slave-95hh krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:13:42.687944 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34643
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34643
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:42.772356 1364 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:42.772562 1364 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:42.772584 1364 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:42.773773 1364 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:42.773823 1364 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:42.773836 1364 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:42.773849 1364 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:42.773873 1364 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:42.775398 1364 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34643
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34643
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1364
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:42.775636 1364 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:42.775846 1364 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:42.780870 1369 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:42.780864 1372 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:42.780869 1370 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:42.780903 1364 server_base.cc:1034] running on GCE node
I20260923 23:13:42.781369 1364 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:42.781642 1364 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:42.782816 1364 hybrid_clock.cc:648] HybridClock initialized: now 1790205222782798 us; error 28 us; skew 500 ppm
Sep 23 23:13:42 dist-test-slave-95hh krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:42.790362 1364 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:13:42.790839 1364 webserver.cc:460] Webserver started at http://127.16.122.190:43461/ using document root <none> and password file <none>
I20260923 23:13:42.791075 1364 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:42.791128 1364 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:42.791270 1364 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:42.792349 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "f3653b29c4f043a4811343ee01ec35bf"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "5c754297b0773d28d41d80a891169954"
server_key_iv: "9fe7e471a3dbbd82d8315a886459de03"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.792671 1364 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "f3653b29c4f043a4811343ee01ec35bf"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "5c754297b0773d28d41d80a891169954"
server_key_iv: "9fe7e471a3dbbd82d8315a886459de03"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.793879 1364 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:13:42.794530 1379 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:42.794718 1364 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:42.794812 1364 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "f3653b29c4f043a4811343ee01ec35bf"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "5c754297b0773d28d41d80a891169954"
server_key_iv: "9fe7e471a3dbbd82d8315a886459de03"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.794875 1364 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:42.813437 1364 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:42.815344 1364 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:42.815483 1364 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:42.819321 1364 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34643
I20260923 23:13:42.819355 1431 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34643 every 8 connection(s)
I20260923 23:13:42.819697 1364 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:42.820530 1432 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:42.822751 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1364
I20260923 23:13:42.822763 1432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: Bootstrap starting.
I20260923 23:13:42.822862 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:42.823060 16874 external_mini_cluster.cc:1428] Setting key 765f68bd9a5d1702fe37aa82bb3cb37e
I20260923 23:13:42.823427 1432 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:42.823801 1432 log.cc:824] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:42.824546 1432 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: No bootstrap required, opened a new log
I20260923 23:13:42.826789 1432 raft_consensus.cc:348] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } }
I20260923 23:13:42.826938 1432 raft_consensus.cc:374] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:42.826960 1432 raft_consensus.cc:729] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f3653b29c4f043a4811343ee01ec35bf, State: Initialized, Role: FOLLOWER
I20260923 23:13:42.827086 1432 consensus_queue.cc:260] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } }
I20260923 23:13:42.827162 1432 raft_consensus.cc:388] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:42.827191 1432 raft_consensus.cc:482] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:42.827247 1432 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:42.827890 1432 raft_consensus.cc:504] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } }
I20260923 23:13:42.827994 1432 leader_election.cc:302] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f3653b29c4f043a4811343ee01ec35bf; no voters:
I20260923 23:13:42.828147 1432 leader_election.cc:288] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:42.828218 1437 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:42.828395 1437 raft_consensus.cc:686] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [term 1 LEADER]: Becoming Leader. State: Replica: f3653b29c4f043a4811343ee01ec35bf, State: Running, Role: LEADER
I20260923 23:13:42.828409 1432 sys_catalog.cc:564] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:42.828545 1437 consensus_queue.cc:237] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } }
I20260923 23:13:42.829046 1438 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f3653b29c4f043a4811343ee01ec35bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } } }
I20260923 23:13:42.829123 1438 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:42.829315 1446 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:42.829339 1439 sys_catalog.cc:455] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader f3653b29c4f043a4811343ee01ec35bf. Latest consensus state: current_term: 1 leader_uuid: "f3653b29c4f043a4811343ee01ec35bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f3653b29c4f043a4811343ee01ec35bf" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34643 } } }
I20260923 23:13:42.829468 1439 sys_catalog.cc:458] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:42.829856 1446 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:13:42 dist-test-slave-95hh krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205222, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:42.831476 1446 catalog_manager.cc:1347] Generated new cluster ID: b01ce43ff587414c8fb63449a160b5a9
I20260923 23:13:42.831527 1446 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:42.834149 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:42.823559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42150 (local address 127.16.122.190:34643)
0923 23:13:42.823771 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:42.823780 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:42.823816 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:13:42.824317 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:42.824326 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:42.824432 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:42.824579 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:42.825183 (+ 604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:42.826637 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:42.827613 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:42.828236 (+ 623us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:42.828879 (+ 643us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:42.831235 (+ 2356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:42.831239 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:42.831240 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:42.831251 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:42.832778 (+ 1527us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:42.833088 (+ 310us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:42.833091 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:42.833091 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:42.833176 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:42.833345 (+ 169us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:42.833346 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:42.833346 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:42.833573 (+ 227us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:42.833727 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:42.833819 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:13:42.833945 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":149,"thread_start_us":94,"threads_started":1}
I20260923 23:13:42.849785 1446 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:42.850307 1446 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:42.855250 1446 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: Generated new TSK 0
I20260923 23:13:42.855430 1446 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:13:42.884438 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:42.963966 1458 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:42.964169 1458 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:42.964191 1458 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:42.965335 1458 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:42.965377 1458 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:42.965402 1458 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:42.966857 1458 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1458
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:42.967106 1458 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:42.967336 1458 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:42.972386 1464 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:42.972342 1463 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:42.972342 1458 server_base.cc:1034] running on GCE node
W20260923 23:13:42.972359 1466 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:42.972779 1458 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:42.973016 1458 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:42.974160 1458 hybrid_clock.cc:648] HybridClock initialized: now 1790205222974151 us; error 28 us; skew 500 ppm
Sep 23 23:13:42 dist-test-slave-95hh krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:42.976538 1458 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:13:42.976951 1458 webserver.cc:460] Webserver started at http://127.16.122.129:45039/ using document root <none> and password file <none>
I20260923 23:13:42.977208 1458 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:42.977255 1458 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:42.977345 1458 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:42.978348 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "1dedee900324496ab1ab484669864066"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "357c92cf5e894c542b6d7d218458abbd"
server_key_iv: "e7dee1f2b18aace97d2b55f5b12ae8f4"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.978650 1458 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "1dedee900324496ab1ab484669864066"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "357c92cf5e894c542b6d7d218458abbd"
server_key_iv: "e7dee1f2b18aace97d2b55f5b12ae8f4"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.979894 1458 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:42.980538 1473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:42.980687 1458 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:42.980772 1458 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "1dedee900324496ab1ab484669864066"
format_stamp: "Formatted at 2026-09-23 23:13:42 on dist-test-slave-95hh"
server_key: "357c92cf5e894c542b6d7d218458abbd"
server_key_iv: "e7dee1f2b18aace97d2b55f5b12ae8f4"
server_key_version: "encryptionkey@0"
I20260923 23:13:42.980831 1458 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.004187 1458 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.006052 1458 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.006167 1458 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.006359 1458 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:43.006619 1458 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:43.006649 1458 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.006669 1458 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:43.006685 1458 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.012446 1458 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:44567
I20260923 23:13:43.012488 1586 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:44567 every 8 connection(s)
I20260923 23:13:43.012795 1458 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
Sep 23 23:13:43 dist-test-slave-95hh krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205222, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:43.019065 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1458
I20260923 23:13:43.019181 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:43.019359 16874 external_mini_cluster.cc:1428] Setting key 1f56b8e574a3667e0147570bae728197
I20260923 23:13:43.021728 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.013469 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:48039 (local address 127.16.122.190:34643)
0923 23:13:43.013512 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.013514 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.013713 (+ 199us) server_negotiation.cc:407] Connection header received
0923 23:13:43.014228 (+ 515us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.014230 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.014243 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.014301 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.015277 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.015779 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.016766 (+ 987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.016976 (+ 210us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.017030 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.019072 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.019073 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.019073 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:43.019080 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.020422 (+ 1342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.020815 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.020816 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.020816 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.020870 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.021093 (+ 223us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.021094 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.021094 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.021211 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.021266 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.021530 (+ 264us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.021635 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:13:43.021894 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.013434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34643 (local address 127.16.122.129:48039)
0923 23:13:43.013600 (+ 166us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.013635 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.014121 (+ 486us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.014384 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.014389 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.014961 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:43.015180 (+ 219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.015186 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.015861 (+ 675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.015862 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.016678 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.016680 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.017044 (+ 364us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.017046 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.017174 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.017620 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:43.017627 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:43.018944 (+ 1317us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:43.020502 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.020510 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.020512 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.020749 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.020924 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.020925 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.020925 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.021039 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.021264 (+ 225us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:43.021271 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:43.021413 (+ 142us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.021502 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.021675 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":125,"thread_start_us":86,"threads_started":1}
I20260923 23:13:43.022351 1587 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34643
I20260923 23:13:43.022457 1587 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:43.022642 1587 heartbeater.cc:511] Master 127.16.122.190:34643 requested a full tablet report, sending...
I20260923 23:13:43.023224 1396 ts_manager.cc:194] Registered new tserver with Master: 1dedee900324496ab1ab484669864066 (127.16.122.129:44567)
I20260923 23:13:43.024113 1396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:48039
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:13:43.043848 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:43.123103 1592 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:43.123328 1592 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:43.123353 1592 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:43.124521 1592 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:43.124562 1592 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:43.124586 1592 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:43.126041 1592 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1592
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:43.126253 1592 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:43.126443 1592 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:43.131459 1597 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.131557 1592 server_base.cc:1034] running on GCE node
W20260923 23:13:43.131467 1598 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.131474 1600 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.131857 1592 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:43.132082 1592 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:43.133224 1592 hybrid_clock.cc:648] HybridClock initialized: now 1790205223133212 us; error 27 us; skew 500 ppm
Sep 23 23:13:43 dist-test-slave-95hh krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:43.135615 1592 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:13:43.136090 1592 webserver.cc:460] Webserver started at http://127.16.122.130:35323/ using document root <none> and password file <none>
I20260923 23:13:43.136322 1592 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:43.136370 1592 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:43.136494 1592 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:43.137552 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "e387ea1b200b471b8f40eabb11557894"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5c5e763f017a809f169149b1912fc56d"
server_key_iv: "0332362b2288a14593e981e5175d5c9c"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.137879 1592 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "e387ea1b200b471b8f40eabb11557894"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5c5e763f017a809f169149b1912fc56d"
server_key_iv: "0332362b2288a14593e981e5175d5c9c"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.139149 1592 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.001s
I20260923 23:13:43.139810 1607 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.139961 1592 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.140040 1592 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "e387ea1b200b471b8f40eabb11557894"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5c5e763f017a809f169149b1912fc56d"
server_key_iv: "0332362b2288a14593e981e5175d5c9c"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.140106 1592 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.149747 1592 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.151571 1592 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.151700 1592 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.151901 1592 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:43.152175 1592 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:43.152207 1592 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.152235 1592 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:43.152256 1592 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.157611 1592 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:38577
I20260923 23:13:43.157678 1720 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:38577 every 8 connection(s)
I20260923 23:13:43.157992 1592 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:43.158243 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1592
I20260923 23:13:43.158344 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:43.158529 16874 external_mini_cluster.cc:1428] Setting key 76745c152b50aab53cbb639bbb05ef47
Sep 23 23:13:43 dist-test-slave-95hh krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
I20260923 23:13:43.167943 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.158828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:45731 (local address 127.16.122.190:34643)
0923 23:13:43.158869 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.158870 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.159355 (+ 485us) server_negotiation.cc:407] Connection header received
0923 23:13:43.159984 (+ 629us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.159985 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.159998 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.160095 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.161189 (+ 1094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.161910 (+ 721us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.162910 (+ 1000us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.163229 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.163300 (+ 71us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.165454 (+ 2154us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.165455 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.165456 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:43.165463 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.166856 (+ 1393us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.167234 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.167235 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.167235 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.167285 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.167453 (+ 168us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.167454 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.167454 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.167545 (+ 91us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.167588 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.167804 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.167873 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:13:43.168183 1723 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.158768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34643 (local address 127.16.122.130:45731)
0923 23:13:43.159219 (+ 451us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.159268 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.159863 (+ 595us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.160176 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.160180 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.160902 (+ 722us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:43.161105 (+ 203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.161110 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.161988 (+ 878us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.161989 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.162833 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.162835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.163300 (+ 465us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.163301 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.163402 (+ 101us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.163916 (+ 514us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:43.163924 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:43.165365 (+ 1441us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:43.166923 (+ 1558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.166930 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.166932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.167169 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.167330 (+ 161us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.167330 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.167330 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.167418 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.167585 (+ 167us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:43.167588 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:43.167704 (+ 116us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.167788 (+ 84us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.167936 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":78,"threads_started":1}
I20260923 23:13:43.168630 1721 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34643
I20260923 23:13:43.168731 1721 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:43.168896 1721 heartbeater.cc:511] Master 127.16.122.190:34643 requested a full tablet report, sending...
I20260923 23:13:43.169281 1396 ts_manager.cc:194] Registered new tserver with Master: e387ea1b200b471b8f40eabb11557894 (127.16.122.130:38577)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
I20260923 23:13:43.169660 1396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:45731
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:13:43.183688 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:43.263401 1726 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:43.263614 1726 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:43.263638 1726 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:43.264797 1726 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:43.264846 1726 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:43.264878 1726 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:43.266376 1726 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34643
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1726
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:43.266600 1726 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:43.266834 1726 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:43.271625 1732 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.271687 1726 server_base.cc:1034] running on GCE node
W20260923 23:13:43.271607 1731 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.271607 1734 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.272017 1726 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:43.272248 1726 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:43.273393 1726 hybrid_clock.cc:648] HybridClock initialized: now 1790205223273380 us; error 29 us; skew 500 ppm
Sep 23 23:13:43 dist-test-slave-95hh krb5kdc[1351](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:43.275749 1726 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:13:43.276171 1726 webserver.cc:460] Webserver started at http://127.16.122.131:39183/ using document root <none> and password file <none>
I20260923 23:13:43.276377 1726 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:43.276424 1726 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:43.276543 1726 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:43.277568 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "4455098f026543e5ad55e251a2a439aa"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "64a8148107cc4873bbabe29964590b1b"
server_key_iv: "4dd29f4220fbaafeeb6b70297d68289f"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.277897 1726 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "4455098f026543e5ad55e251a2a439aa"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "64a8148107cc4873bbabe29964590b1b"
server_key_iv: "4dd29f4220fbaafeeb6b70297d68289f"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.279096 1726 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:13:43.279759 1741 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.279902 1726 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:43.279979 1726 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "4455098f026543e5ad55e251a2a439aa"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "64a8148107cc4873bbabe29964590b1b"
server_key_iv: "4dd29f4220fbaafeeb6b70297d68289f"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.280042 1726 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.289932 1726 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.291838 1726 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.291962 1726 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.292161 1726 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:43.292425 1726 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:43.292457 1726 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.292483 1726 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:43.292512 1726 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.298060 1726 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:34591
I20260923 23:13:43.298108 1854 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:34591 every 8 connection(s)
I20260923 23:13:43.298424 1726 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:13:43 dist-test-slave-95hh krb5kdc[1351](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205223, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:43.306288 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.299103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:41789 (local address 127.16.122.190:34643)
0923 23:13:43.299147 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.299149 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.299341 (+ 192us) server_negotiation.cc:407] Connection header received
0923 23:13:43.299802 (+ 461us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.299803 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.299812 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.299860 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.300729 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.301217 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.302124 (+ 907us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.302299 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.302340 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.304206 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.304207 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.304208 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:43.304215 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.305275 (+ 1060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.305597 (+ 322us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.305598 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.305598 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.305643 (+ 45us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.305822 (+ 179us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.305822 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.305822 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.305918 (+ 96us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.305960 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.306161 (+ 201us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.306219 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:13:43.306473 1857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.299086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34643 (local address 127.16.122.131:41789)
0923 23:13:43.299226 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.299262 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.299728 (+ 466us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.299916 (+ 188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.299919 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.300465 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:43.300667 (+ 202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.300671 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.301280 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.301280 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.302067 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.302069 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.302352 (+ 283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.302352 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.302444 (+ 92us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.302842 (+ 398us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:43.302849 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:43.304112 (+ 1263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:43.305337 (+ 1225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.305342 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.305343 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.305542 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.305694 (+ 152us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:43.305695 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:43.305695 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:43.305782 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:43.305957 (+ 175us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:43.305960 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:43.306068 (+ 108us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.306145 (+ 77us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.306262 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":79,"threads_started":1}
I20260923 23:13:43.306919 1855 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34643
I20260923 23:13:43.306998 1855 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:43.307195 1855 heartbeater.cc:511] Master 127.16.122.190:34643 requested a full tablet report, sending...
I20260923 23:13:43.307529 1396 ts_manager.cc:194] Registered new tserver with Master: 4455098f026543e5ad55e251a2a439aa (127.16.122.131:34591)
I20260923 23:13:43.307916 1396 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:41789
I20260923 23:13:43.308097 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1726
I20260923 23:13:43.308207 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:43.308395 16874 external_mini_cluster.cc:1428] Setting key 4e823eab2de662599181c8b34e732131
I20260923 23:13:43.309142 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:43.315771 1435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.310402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42154 (local address 127.16.122.190:34643)
0923 23:13:43.310438 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.310439 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.310540 (+ 101us) server_negotiation.cc:407] Connection header received
0923 23:13:43.310603 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.310603 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.310611 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.310643 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.311207 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.311697 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.312356 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.312531 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.312577 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.313619 (+ 1042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.313620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.313620 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:43.313627 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.314668 (+ 1041us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.314946 (+ 278us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.314947 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.314947 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.315021 (+ 74us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:43.315238 (+ 217us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:43.315239 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:43.315239 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:43.315382 (+ 143us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.315471 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.315608 (+ 137us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.315685 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:13:43.317332 1396 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42154:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:43.318945 1396 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:43.325011 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.321321 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40246 (local address 127.16.122.129:44567)
0923 23:13:43.321478 (+ 157us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.321481 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.321491 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:13:43.321526 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.321528 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.321541 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.321623 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:43.321824 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322867 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324156 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324630 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324699 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324758 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.324879 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.324926 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":80,"threads_started":1}
I20260923 23:13:43.324995 1863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.321109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:44567 (local address 127.0.0.1:40246)
0923 23:13:43.321330 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.321339 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.321380 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.321700 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.321700 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.321702 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:43.321755 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.321756 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322961 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.322961 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.324081 (+ 1120us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.324083 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324704 (+ 621us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324705 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.324780 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324794 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.324858 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.324913 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":211,"mutex_wait_us":62,"thread_start_us":69,"threads_started":1}
I20260923 23:13:43.325129 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.320979 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49648 (local address 127.16.122.130:38577)
0923 23:13:43.321159 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.321164 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.321279 (+ 115us) server_negotiation.cc:407] Connection header received
0923 23:13:43.321534 (+ 255us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.321537 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.321550 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.321614 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:43.321852 (+ 238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322556 (+ 704us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.323938 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324678 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324750 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324807 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.325003 (+ 196us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.325047 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":85,"threads_started":1}
I20260923 23:13:43.325165 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.320988 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38024 (local address 127.16.122.131:34591)
0923 23:13:43.321236 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.321239 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.321287 (+ 48us) server_negotiation.cc:407] Connection header received
0923 23:13:43.321386 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.321389 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.321404 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.321472 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:43.321702 (+ 230us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322822 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.323955 (+ 1133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324725 (+ 770us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324778 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324820 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.325041 (+ 221us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.325094 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":218,"thread_start_us":80,"threads_started":1}
I20260923 23:13:43.325258 1867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.320893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:34591 (local address 127.0.0.1:38024)
0923 23:13:43.321234 (+ 341us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.321244 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.321312 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.321558 (+ 246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.321563 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.321565 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:43.321629 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.321631 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322948 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.322949 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.323877 (+ 928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.323879 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324847 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324848 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.324910 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324916 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.324961 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.325001 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":320,"mutex_wait_us":32,"thread_start_us":83,"threads_started":1}
I20260923 23:13:43.325367 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.320889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:38577 (local address 127.0.0.1:49648)
0923 23:13:43.321172 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.321215 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.321428 (+ 213us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.321618 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.321619 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.321620 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:43.321767 (+ 147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.321770 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.322650 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.322651 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.323853 (+ 1202us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.323856 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.324843 (+ 987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.324844 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.324910 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.324916 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.325204 (+ 288us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.325295 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":243,"mutex_wait_us":2,"thread_start_us":83,"threads_started":1}
I20260923 23:13:43.326071 1789 tablet_service.cc:1467] Processing CreateTablet for tablet c3649afa569d464fa2c01db39eea6e90 (DEFAULT_TABLE table=test-table [id=5eceed98ade44c30996040169421ceee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:43.326153 1655 tablet_service.cc:1467] Processing CreateTablet for tablet c3649afa569d464fa2c01db39eea6e90 (DEFAULT_TABLE table=test-table [id=5eceed98ade44c30996040169421ceee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:43.326342 1520 tablet_service.cc:1467] Processing CreateTablet for tablet c3649afa569d464fa2c01db39eea6e90 (DEFAULT_TABLE table=test-table [id=5eceed98ade44c30996040169421ceee]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:43.326443 1789 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3649afa569d464fa2c01db39eea6e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:43.326442 1655 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3649afa569d464fa2c01db39eea6e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:43.326644 1520 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet c3649afa569d464fa2c01db39eea6e90. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:43.329439 1869 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: Bootstrap starting.
I20260923 23:13:43.329840 1870 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Bootstrap starting.
I20260923 23:13:43.330060 1871 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Bootstrap starting.
I20260923 23:13:43.330274 1869 tablet_bootstrap.cc:654] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:43.330706 1871 tablet_bootstrap.cc:654] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:43.330879 1869 log.cc:824] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:43.331265 1871 log.cc:824] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:43.331645 1870 tablet_bootstrap.cc:654] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:43.331697 1869 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: No bootstrap required, opened a new log
I20260923 23:13:43.331792 1869 ts_tablet_manager.cc:1397] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:43.331923 1871 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: No bootstrap required, opened a new log
I20260923 23:13:43.331985 1871 ts_tablet_manager.cc:1397] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:43.332180 1870 log.cc:824] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:43.332902 1870 tablet_bootstrap.cc:492] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: No bootstrap required, opened a new log
I20260923 23:13:43.332973 1870 ts_tablet_manager.cc:1397] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Time spent bootstrapping tablet: real 0.003s user 0.001s sys 0.000s
I20260923 23:13:43.333474 1869 raft_consensus.cc:348] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.333609 1869 raft_consensus.cc:374] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:43.333629 1869 raft_consensus.cc:729] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1dedee900324496ab1ab484669864066, State: Initialized, Role: FOLLOWER
I20260923 23:13:43.333722 1869 consensus_queue.cc:260] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.333971 1587 heartbeater.cc:503] Master 127.16.122.190:34643 was elected leader, sending a full tablet report...
I20260923 23:13:43.333881 1871 raft_consensus.cc:348] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.334076 1871 raft_consensus.cc:374] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:43.334112 1871 raft_consensus.cc:729] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e387ea1b200b471b8f40eabb11557894, State: Initialized, Role: FOLLOWER
I20260923 23:13:43.333961 1869 ts_tablet_manager.cc:1428] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:43.334249 1871 consensus_queue.cc:260] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.334509 1871 ts_tablet_manager.cc:1428] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:43.334522 1721 heartbeater.cc:503] Master 127.16.122.190:34643 was elected leader, sending a full tablet report...
I20260923 23:13:43.334837 1870 raft_consensus.cc:348] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.334983 1870 raft_consensus.cc:374] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:43.335014 1870 raft_consensus.cc:729] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4455098f026543e5ad55e251a2a439aa, State: Initialized, Role: FOLLOWER
I20260923 23:13:43.335158 1870 consensus_queue.cc:260] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.335392 1870 ts_tablet_manager.cc:1428] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:43.335459 1855 heartbeater.cc:503] Master 127.16.122.190:34643 was elected leader, sending a full tablet report...
W20260923 23:13:43.408653 1722 tablet.cc:2369] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:13:43.464277 1875 raft_consensus.cc:482] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:43.464453 1875 raft_consensus.cc:504] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.464846 1875 leader_election.cc:288] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4455098f026543e5ad55e251a2a439aa (127.16.122.131:34591), e387ea1b200b471b8f40eabb11557894 (127.16.122.130:38577)
I20260923 23:13:43.468559 1589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.464995 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:34591 (local address 127.16.122.129:49733)
0923 23:13:43.465058 (+ 63us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.465071 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.465130 (+ 59us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.465278 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.465279 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.465281 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:43.465350 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.465352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.466245 (+ 893us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.466246 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.467591 (+ 1345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.467593 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.468348 (+ 755us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.468349 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.468412 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.468416 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.468458 (+ 42us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.468490 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39,"mutex_wait_us":34}
I20260923 23:13:43.468592 1866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.465116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:49733 (local address 127.16.122.131:34591)
0923 23:13:43.465173 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.465176 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.465183 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:43.465208 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.465209 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.465218 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.465270 (+ 52us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:43.465420 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.466170 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.467686 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.468277 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.468341 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.468350 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.468481 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.468524 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:13:43.468644 1878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.464996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:38577 (local address 127.16.122.129:46501)
0923 23:13:43.465213 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.465223 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.465275 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.465380 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.465381 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.465382 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:43.465440 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.465441 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.466270 (+ 829us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.466271 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.467575 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.467579 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.468401 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.468403 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.468481 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.468485 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.468536 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.468574 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":197,"mutex_wait_us":78,"thread_start_us":125,"threads_started":1}
I20260923 23:13:43.468665 1865 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.465204 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:46501 (local address 127.16.122.130:38577)
0923 23:13:43.465264 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.465266 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.465275 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:43.465324 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.465325 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.465333 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.465377 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:43.465496 (+ 119us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.466211 (+ 715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.467676 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.468267 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.468341 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.468350 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.468557 (+ 207us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.468606 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20260923 23:13:43.468904 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3649afa569d464fa2c01db39eea6e90" candidate_uuid: "1dedee900324496ab1ab484669864066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4455098f026543e5ad55e251a2a439aa" is_pre_election: true
I20260923 23:13:43.468943 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3649afa569d464fa2c01db39eea6e90" candidate_uuid: "1dedee900324496ab1ab484669864066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e387ea1b200b471b8f40eabb11557894" is_pre_election: true
I20260923 23:13:43.469058 1809 raft_consensus.cc:2454] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dedee900324496ab1ab484669864066 in term 0.
I20260923 23:13:43.469070 1675 raft_consensus.cc:2454] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1dedee900324496ab1ab484669864066 in term 0.
I20260923 23:13:43.469272 1474 leader_election.cc:302] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dedee900324496ab1ab484669864066, e387ea1b200b471b8f40eabb11557894; no voters:
I20260923 23:13:43.469429 1875 raft_consensus.cc:2781] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:43.469481 1875 raft_consensus.cc:482] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:43.469496 1875 raft_consensus.cc:3037] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:43.470142 1875 raft_consensus.cc:504] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.470256 1875 leader_election.cc:288] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [CANDIDATE]: Term 1 election: Requested vote from peers 4455098f026543e5ad55e251a2a439aa (127.16.122.131:34591), e387ea1b200b471b8f40eabb11557894 (127.16.122.130:38577)
I20260923 23:13:43.470417 1809 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3649afa569d464fa2c01db39eea6e90" candidate_uuid: "1dedee900324496ab1ab484669864066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4455098f026543e5ad55e251a2a439aa"
I20260923 23:13:43.470492 1675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "c3649afa569d464fa2c01db39eea6e90" candidate_uuid: "1dedee900324496ab1ab484669864066" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e387ea1b200b471b8f40eabb11557894"
I20260923 23:13:43.470492 1809 raft_consensus.cc:3037] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:43.470561 1675 raft_consensus.cc:3037] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:43.471397 1675 raft_consensus.cc:2454] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dedee900324496ab1ab484669864066 in term 1.
I20260923 23:13:43.471397 1809 raft_consensus.cc:2454] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1dedee900324496ab1ab484669864066 in term 1.
I20260923 23:13:43.471558 1475 leader_election.cc:302] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1dedee900324496ab1ab484669864066, 4455098f026543e5ad55e251a2a439aa; no voters:
I20260923 23:13:43.471639 1875 raft_consensus.cc:2781] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:43.471772 1875 raft_consensus.cc:686] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [term 1 LEADER]: Becoming Leader. State: Replica: 1dedee900324496ab1ab484669864066, State: Running, Role: LEADER
I20260923 23:13:43.471851 1875 consensus_queue.cc:237] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } }
I20260923 23:13:43.472673 1396 catalog_manager.cc:5495] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1dedee900324496ab1ab484669864066 (127.16.122.129). New cstate: current_term: 1 leader_uuid: "1dedee900324496ab1ab484669864066" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1dedee900324496ab1ab484669864066" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 44567 } health_report { overall_health: HEALTHY } } }
I20260923 23:13:43.504331 1868 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.502256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40256 (local address 127.16.122.129:44567)
0923 23:13:43.502310 (+ 54us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.502311 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.502318 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:43.502353 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.502353 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.502373 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.502425 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:43.502541 (+ 116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.502994 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.503710 (+ 716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.503888 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.503933 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.504044 (+ 111us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:43.504149 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:43.504203 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.504214 (+ 11us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.504231 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:13:43.506146 1809 raft_consensus.cc:1261] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 1 FOLLOWER]: Refusing update from remote peer 1dedee900324496ab1ab484669864066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:43.506146 1675 raft_consensus.cc:1261] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 1 FOLLOWER]: Refusing update from remote peer 1dedee900324496ab1ab484669864066: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:43.506457 1879 consensus_queue.cc:1035] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4455098f026543e5ad55e251a2a439aa" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 34591 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:43.506589 1875 consensus_queue.cc:1035] T c3649afa569d464fa2c01db39eea6e90 P 1dedee900324496ab1ab484669864066 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e387ea1b200b471b8f40eabb11557894" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38577 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:43.508919 1882 mvcc.cc:205] Tried to move back new op lower bound from 7332680595478556672 to 7332680595341623296. Current Snapshot: MvccSnapshot[applied={T|T < 7332680595478556672}]
I20260923 23:13:43.509012 1884 mvcc.cc:205] Tried to move back new op lower bound from 7332680595478556672 to 7332680595341623296. Current Snapshot: MvccSnapshot[applied={T|T < 7332680595478556672}]
I20260923 23:13:43.510316 1883 mvcc.cc:205] Tried to move back new op lower bound from 7332680595478556672 to 7332680595341623296. Current Snapshot: MvccSnapshot[applied={T|T < 7332680595478556672 or (T in {7332680595478556672})}]
I20260923 23:13:43.511307 1395 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:43.511407 1395 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42154:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:43.512138 1395 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f3653b29c4f043a4811343ee01ec35bf: Sending DeleteTablet for 3 replicas of tablet c3649afa569d464fa2c01db39eea6e90
I20260923 23:13:43.512485 1789 tablet_service.cc:1514] Processing DeleteTablet for tablet c3649afa569d464fa2c01db39eea6e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:43 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:38024
I20260923 23:13:43.512473 1520 tablet_service.cc:1514] Processing DeleteTablet for tablet c3649afa569d464fa2c01db39eea6e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:43 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:40246
I20260923 23:13:43.512498 1655 tablet_service.cc:1514] Processing DeleteTablet for tablet c3649afa569d464fa2c01db39eea6e90 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:43 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49648
I20260923 23:13:43.512713 1893 tablet_replica.cc:330] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: stopping tablet replica
I20260923 23:13:43.512809 1893 raft_consensus.cc:2229] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:43.512850 1892 tablet_replica.cc:330] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: stopping tablet replica
I20260923 23:13:43.512907 1893 raft_consensus.cc:2258] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:43.512939 1892 raft_consensus.cc:2229] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:43.513031 1892 raft_consensus.cc:2258] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:43.513039 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1458
I20260923 23:13:43.513952 1893 ts_tablet_manager.cc:1905] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:43.514380 1892 ts_tablet_manager.cc:1905] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:43.515547 1893 ts_tablet_manager.cc:1918] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:43.515606 1893 log.cc:1196] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/c3649afa569d464fa2c01db39eea6e90
I20260923 23:13:43.515836 1893 ts_tablet_manager.cc:1939] T c3649afa569d464fa2c01db39eea6e90 P e387ea1b200b471b8f40eabb11557894: Deleting consensus metadata
I20260923 23:13:43.515848 1892 ts_tablet_manager.cc:1918] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:43.515904 1892 log.cc:1196] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/c3649afa569d464fa2c01db39eea6e90
I20260923 23:13:43.516149 1892 ts_tablet_manager.cc:1939] T c3649afa569d464fa2c01db39eea6e90 P 4455098f026543e5ad55e251a2a439aa: Deleting consensus metadata
I20260923 23:13:43.516278 1380 catalog_manager.cc:4841] TS e387ea1b200b471b8f40eabb11557894 (127.16.122.130:38577): tablet c3649afa569d464fa2c01db39eea6e90 (table test-table [id=5eceed98ade44c30996040169421ceee]) successfully deleted
I20260923 23:13:43.516489 1381 catalog_manager.cc:4841] TS 4455098f026543e5ad55e251a2a439aa (127.16.122.131:34591): tablet c3649afa569d464fa2c01db39eea6e90 (table test-table [id=5eceed98ade44c30996040169421ceee]) successfully deleted
W20260923 23:13:43.518297 1380 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:44567 (error 108)
I20260923 23:13:43.518528 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1592
I20260923 23:13:43.519109 1864 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.518592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:44567 (local address 127.0.0.1:40262)
0923 23:13:43.519012 (+ 420us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.519075 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:44567: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395}
W20260923 23:13:43.519232 1380 catalog_manager.cc:4584] TS 1dedee900324496ab1ab484669864066 (127.16.122.129:44567): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c3649afa569d464fa2c01db39eea6e90: Network error: Client connection negotiation failed: client connection to 127.16.122.129:44567: connect: Connection refused (error 111)
I20260923 23:13:43.524010 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1726
I20260923 23:13:43.527797 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1364
2026-09-23T23:13:43Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (1898 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-09-23T23:13:43Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:43Z Disabled control of system clock
I20260923 23:13:43.553736 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41295
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:41295
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:43.632025 1898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:43.632253 1898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:43.632273 1898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:43.633437 1898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:43.633481 1898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:43.633499 1898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:43.633512 1898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:43.633522 1898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:43.634873 1898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:41295
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:41295
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1898
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:43.635133 1898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:43.635421 1898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:43.640189 1906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.640196 1903 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.640196 1904 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.640236 1898 server_base.cc:1034] running on GCE node
I20260923 23:13:43.640611 1898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:43.640892 1898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:43.642028 1898 hybrid_clock.cc:648] HybridClock initialized: now 1790205223642014 us; error 30 us; skew 500 ppm
I20260923 23:13:43.643218 1898 webserver.cc:460] Webserver started at http://127.16.122.190:41741/ using document root <none> and password file <none>
I20260923 23:13:43.643433 1898 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:43.643481 1898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:43.643615 1898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:43.644665 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "0232c4c778d643019f02127f880fb456"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5e5b7c5c5467b8000b5f0c0632c8cf64"
server_key_iv: "49fe3dc7e025b1204d3d9eb20d4e2e38"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.644995 1898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "0232c4c778d643019f02127f880fb456"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5e5b7c5c5467b8000b5f0c0632c8cf64"
server_key_iv: "49fe3dc7e025b1204d3d9eb20d4e2e38"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.646198 1898 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:43.646867 1912 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.647011 1898 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:43.647128 1898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "0232c4c778d643019f02127f880fb456"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "5e5b7c5c5467b8000b5f0c0632c8cf64"
server_key_iv: "49fe3dc7e025b1204d3d9eb20d4e2e38"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.647195 1898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.685261 1898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.685603 1898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.685750 1898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.689687 1898 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:41295
I20260923 23:13:43.689719 1964 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:41295 every 8 connection(s)
I20260923 23:13:43.690057 1898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:43.691051 1965 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:43.693612 1965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: Bootstrap starting.
I20260923 23:13:43.694195 1965 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:43.694433 1965 log.cc:824] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:43.695201 1965 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: No bootstrap required, opened a new log
I20260923 23:13:43.696767 1965 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } }
I20260923 23:13:43.696928 1965 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:43.696957 1965 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0232c4c778d643019f02127f880fb456, State: Initialized, Role: FOLLOWER
I20260923 23:13:43.697070 1965 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } }
I20260923 23:13:43.697120 1965 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:43.697139 1965 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:43.697168 1965 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:43.697657 1965 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } }
I20260923 23:13:43.697739 1965 leader_election.cc:302] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0232c4c778d643019f02127f880fb456; no voters:
I20260923 23:13:43.697867 1965 leader_election.cc:288] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:43.697925 1968 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:43.698110 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1898
I20260923 23:13:43.698118 1965 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:43.698158 1968 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [term 1 LEADER]: Becoming Leader. State: Replica: 0232c4c778d643019f02127f880fb456, State: Running, Role: LEADER
I20260923 23:13:43.698206 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:43.698253 1968 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } }
I20260923 23:13:43.698410 16874 external_mini_cluster.cc:1428] Setting key 747156767e4d922a2175262c18e2e54e
I20260923 23:13:43.698702 1969 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0232c4c778d643019f02127f880fb456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } } }
I20260923 23:13:43.698787 1969 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:43.698941 1970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0232c4c778d643019f02127f880fb456. Latest consensus state: current_term: 1 leader_uuid: "0232c4c778d643019f02127f880fb456" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0232c4c778d643019f02127f880fb456" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 41295 } } }
I20260923 23:13:43.698998 1970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456 [sys.catalog]: This master's current role is: LEADER
W20260923 23:13:43.699784 1985 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:13:43.699828 1985 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:13:43.699884 1978 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:43.700049 1978 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:43.701514 1978 catalog_manager.cc:1347] Generated new cluster ID: 3c415ba73b61468b8c01472709caa204
I20260923 23:13:43.701565 1978 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:43.704728 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.699085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.16.122.190:41295)
0923 23:13:43.699262 (+ 177us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.699271 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.699355 (+ 84us) server_negotiation.cc:407] Connection header received
0923 23:13:43.699981 (+ 626us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.699988 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.700064 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.700219 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.700447 (+ 228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.701595 (+ 1148us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.702670 (+ 1075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.703290 (+ 620us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.703397 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.704057 (+ 660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.704061 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.704062 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:43.704262 (+ 200us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.704301 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:43.704315 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.704369 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.704454 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.704524 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95,"thread_start_us":73,"threads_started":1}
I20260923 23:13:43.728786 1978 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:43.729519 1978 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:43.739639 1978 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: Generated new TSK 0
I20260923 23:13:43.739872 1978 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:43.742517 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--builtin_ntp_servers=127.16.122.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:43.821714 1989 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:43.821921 1989 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:43.821951 1989 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:43.823122 1989 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:43.823159 1989 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:43.823192 1989 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:43.824504 1989 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.1989
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:43.824730 1989 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:43.825003 1989 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:43.829820 1995 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.829799 1994 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.829919 1989 server_base.cc:1034] running on GCE node
W20260923 23:13:43.829789 1997 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.830220 1989 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:43.830428 1989 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:43.831552 1989 hybrid_clock.cc:648] HybridClock initialized: now 1790205223831540 us; error 27 us; skew 500 ppm
I20260923 23:13:43.832746 1989 webserver.cc:460] Webserver started at http://127.16.122.129:33283/ using document root <none> and password file <none>
I20260923 23:13:43.832964 1989 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:43.833009 1989 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:43.833127 1989 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:43.834156 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "5eff68d4d2c2406c9a999afe173e918f"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "a09123ccb686cdc1546068d780aee6ce"
server_key_iv: "da1acbf01106d16f4d8ab22578edb57b"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.834478 1989 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "5eff68d4d2c2406c9a999afe173e918f"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "a09123ccb686cdc1546068d780aee6ce"
server_key_iv: "da1acbf01106d16f4d8ab22578edb57b"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.835711 1989 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:13:43.836333 2003 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.836479 1989 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:43.836583 1989 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "5eff68d4d2c2406c9a999afe173e918f"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "a09123ccb686cdc1546068d780aee6ce"
server_key_iv: "da1acbf01106d16f4d8ab22578edb57b"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.836663 1989 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.849526 1989 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.849821 1989 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.849939 1989 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.850137 1989 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:43.850430 1989 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:43.850461 1989 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.850481 1989 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:43.850503 1989 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.855839 1989 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:40867
I20260923 23:13:43.855894 2116 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:40867 every 8 connection(s)
I20260923 23:13:43.856179 1989 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:43.856966 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 1989
I20260923 23:13:43.857067 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:43.857246 16874 external_mini_cluster.cc:1428] Setting key 8abb09e69cace7eb7e4a42fdaa84cce4
I20260923 23:13:43.858075 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--builtin_ntp_servers=127.16.122.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:43.861699 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.856794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:40079 (local address 127.16.122.190:41295)
0923 23:13:43.856841 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.856843 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.857088 (+ 245us) server_negotiation.cc:407] Connection header received
0923 23:13:43.857661 (+ 573us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.857662 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.857676 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.857711 (+ 35us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.858460 (+ 749us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.858984 (+ 524us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.860375 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.860581 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.860629 (+ 48us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.861459 (+ 830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.861460 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.861460 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:43.861464 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.861479 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:43.861491 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.861529 (+ 38us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.861609 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.861640 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:13:43.861954 2119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.856794 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41295 (local address 127.16.122.129:40079)
0923 23:13:43.856973 (+ 179us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.857007 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.857566 (+ 559us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.857796 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.857800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.857810 (+ 10us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:43.858379 (+ 569us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.858386 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.859119 (+ 733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.859120 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.860279 (+ 1159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.860283 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.860650 (+ 367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.860650 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.860795 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.861313 (+ 518us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:43.861314 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:43.861353 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:43.861381 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:43.861531 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:43.861535 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:43.861538 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.861614 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.861726 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":121,"thread_start_us":83,"threads_started":1}
I20260923 23:13:43.862408 2117 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41295
I20260923 23:13:43.862493 2117 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:43.862664 2117 heartbeater.cc:511] Master 127.16.122.190:41295 requested a full tablet report, sending...
I20260923 23:13:43.863162 1929 ts_manager.cc:194] Registered new tserver with Master: 5eff68d4d2c2406c9a999afe173e918f (127.16.122.129:40867)
I20260923 23:13:43.863751 1929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.129:40079
W20260923 23:13:43.939445 2120 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:43.939654 2120 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:43.939683 2120 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:43.940943 2120 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:43.940987 2120 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:43.941021 2120 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:43.942375 2120 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2120
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:43.942613 2120 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:43.942889 2120 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:43.947688 2126 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.947855 2128 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:43.947739 2125 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:43.947769 2120 server_base.cc:1034] running on GCE node
I20260923 23:13:43.948149 2120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:43.948374 2120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:43.949515 2120 hybrid_clock.cc:648] HybridClock initialized: now 1790205223949502 us; error 32 us; skew 500 ppm
I20260923 23:13:43.950757 2120 webserver.cc:460] Webserver started at http://127.16.122.130:35077/ using document root <none> and password file <none>
I20260923 23:13:43.951000 2120 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:43.951083 2120 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:43.951215 2120 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:43.952268 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "a69d06162043464cae35a23693f4d272"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "d686f190e93aa92e162f5e810c8b8588"
server_key_iv: "d22de05d0be2f4f4297b31d6273c1ed2"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.952598 2120 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "a69d06162043464cae35a23693f4d272"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "d686f190e93aa92e162f5e810c8b8588"
server_key_iv: "d22de05d0be2f4f4297b31d6273c1ed2"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.953806 2120 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:13:43.954434 2134 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.954589 2120 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:43.954668 2120 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "a69d06162043464cae35a23693f4d272"
format_stamp: "Formatted at 2026-09-23 23:13:43 on dist-test-slave-95hh"
server_key: "d686f190e93aa92e162f5e810c8b8588"
server_key_iv: "d22de05d0be2f4f4297b31d6273c1ed2"
server_key_version: "encryptionkey@0"
I20260923 23:13:43.954736 2120 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:43.964052 2120 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:43.964322 2120 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:43.964438 2120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:43.964653 2120 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:43.964980 2120 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:43.965013 2120 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.965042 2120 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:43.965065 2120 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:43.970291 2120 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:34857
I20260923 23:13:43.970350 2247 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:34857 every 8 connection(s)
I20260923 23:13:43.970645 2120 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:43.973165 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2120
I20260923 23:13:43.973244 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:43.973444 16874 external_mini_cluster.cc:1428] Setting key fcacdbbac31083043c0574ab26a1afa2
I20260923 23:13:43.974418 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--builtin_ntp_servers=127.16.122.148:39783
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:43.975960 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.971314 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:54047 (local address 127.16.122.190:41295)
0923 23:13:43.971369 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:43.971370 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:43.971584 (+ 214us) server_negotiation.cc:407] Connection header received
0923 23:13:43.972062 (+ 478us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:43.972063 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:43.972074 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:43.972120 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:43.972723 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.973201 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.974580 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.974784 (+ 204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.974834 (+ 50us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.975688 (+ 854us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:43.975689 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:43.975690 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:43.975695 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:43.975714 (+ 19us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:43.975718 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:43.975772 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:43.975848 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 23:13:43.975878 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:13:43.976109 2250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:43.971313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41295 (local address 127.16.122.130:54047)
0923 23:13:43.971473 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:43.971506 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:43.971975 (+ 469us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:43.972177 (+ 202us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:43.972180 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:43.972188 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:43.972665 (+ 477us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.972669 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.973285 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.973286 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.974411 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:43.974415 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:43.974855 (+ 440us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:43.974856 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:43.975008 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:43.975528 (+ 520us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:43.975528 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:43.975569 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:43.975598 (+ 29us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:43.975779 (+ 181us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:43.975783 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:43.975786 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:43.975841 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 23:13:43.975919 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":94,"thread_start_us":65,"threads_started":1}
I20260923 23:13:43.976526 2248 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41295
I20260923 23:13:43.976610 2248 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:43.976783 2248 heartbeater.cc:511] Master 127.16.122.190:41295 requested a full tablet report, sending...
I20260923 23:13:43.977137 1929 ts_manager.cc:194] Registered new tserver with Master: a69d06162043464cae35a23693f4d272 (127.16.122.130:34857)
I20260923 23:13:43.977496 1929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.130:54047
W20260923 23:13:44.054239 2251 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:44.054469 2251 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:44.054500 2251 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:44.055707 2251 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:44.055747 2251 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:44.055779 2251 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:44.057166 2251 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:39783
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:41295
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2251
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:44.057407 2251 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:44.057693 2251 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:44.062490 2259 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:44.062501 2256 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:44.062691 2251 server_base.cc:1034] running on GCE node
W20260923 23:13:44.062502 2257 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:44.062974 2251 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:44.063257 2251 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:44.064391 2251 hybrid_clock.cc:648] HybridClock initialized: now 1790205224064380 us; error 28 us; skew 500 ppm
I20260923 23:13:44.065613 2251 webserver.cc:460] Webserver started at http://127.16.122.131:45227/ using document root <none> and password file <none>
I20260923 23:13:44.065826 2251 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:44.065869 2251 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:44.066007 2251 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:44.067102 2251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "9270b204885b4219bd9e5e63712ea9bb"
format_stamp: "Formatted at 2026-09-23 23:13:44 on dist-test-slave-95hh"
server_key: "668fc95c130c4cf8ae4d4328a8e84e99"
server_key_iv: "d8d215d44eab2ea0536c37697230f9dd"
server_key_version: "encryptionkey@0"
I20260923 23:13:44.067435 2251 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "9270b204885b4219bd9e5e63712ea9bb"
format_stamp: "Formatted at 2026-09-23 23:13:44 on dist-test-slave-95hh"
server_key: "668fc95c130c4cf8ae4d4328a8e84e99"
server_key_iv: "d8d215d44eab2ea0536c37697230f9dd"
server_key_version: "encryptionkey@0"
I20260923 23:13:44.068601 2251 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:13:44.069208 2265 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:44.069352 2251 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:44.069432 2251 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "9270b204885b4219bd9e5e63712ea9bb"
format_stamp: "Formatted at 2026-09-23 23:13:44 on dist-test-slave-95hh"
server_key: "668fc95c130c4cf8ae4d4328a8e84e99"
server_key_iv: "d8d215d44eab2ea0536c37697230f9dd"
server_key_version: "encryptionkey@0"
I20260923 23:13:44.069496 2251 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:44.104660 2251 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:44.104981 2251 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:44.105113 2251 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:44.105329 2251 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:44.105664 2251 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:44.105697 2251 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:44.105727 2251 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:44.105751 2251 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:44.111094 2251 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:33101
I20260923 23:13:44.111142 2378 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:33101 every 8 connection(s)
I20260923 23:13:44.111459 2251 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:13:44.116183 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.112151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:45501 (local address 127.16.122.190:41295)
0923 23:13:44.112204 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.112206 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.112456 (+ 250us) server_negotiation.cc:407] Connection header received
0923 23:13:44.112938 (+ 482us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.112939 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.112949 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.112999 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:44.113626 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.114114 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.115023 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.115211 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.115254 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.115974 (+ 720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:44.115975 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:44.115975 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:44.115979 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:44.115992 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:44.115995 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:44.116036 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.116096 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.116121 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:13:44.116377 2381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.112148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:41295 (local address 127.16.122.131:45501)
0923 23:13:44.112331 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.112367 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.112860 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.113060 (+ 200us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.113063 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.113072 (+ 9us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:44.113563 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.113568 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.114177 (+ 609us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.114178 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.114961 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.114963 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.115269 (+ 306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.115269 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.115380 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.115852 (+ 472us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:44.115853 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:44.115881 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:44.115903 (+ 22us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:44.116041 (+ 138us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:44.116044 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:44.116046 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.116091 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.116169 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":87,"threads_started":1}
I20260923 23:13:44.116813 2379 heartbeater.cc:347] Connected to a master server at 127.16.122.190:41295
I20260923 23:13:44.116904 2379 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:44.117082 2379 heartbeater.cc:511] Master 127.16.122.190:41295 requested a full tablet report, sending...
I20260923 23:13:44.117426 1929 ts_manager.cc:194] Registered new tserver with Master: 9270b204885b4219bd9e5e63712ea9bb (127.16.122.131:33101)
I20260923 23:13:44.117750 1929 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.131:45501
I20260923 23:13:44.120060 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2251
I20260923 23:13:44.120142 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:44.120298 16874 external_mini_cluster.cc:1428] Setting key 4ca5e376392666d28467690282c264b3
I20260923 23:13:44.121109 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:44.124864 1981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.122391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43014 (local address 127.16.122.190:41295)
0923 23:13:44.122440 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.122442 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.122536 (+ 94us) server_negotiation.cc:407] Connection header received
0923 23:13:44.122588 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.122588 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.122597 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.122633 (+ 36us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:44.122758 (+ 125us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.123275 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.123957 (+ 682us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.124129 (+ 172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.124170 (+ 41us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.124684 (+ 514us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:44.124685 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:44.124685 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:44.124689 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:44.124701 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:44.124704 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:44.124741 (+ 37us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.124784 (+ 43us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.124806 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:13:44.126181 1929 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:43014:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:44.127764 1929 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:44.133661 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.129736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34128 (local address 127.16.122.129:40867)
0923 23:13:44.130166 (+ 430us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.130170 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.130178 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:44.130215 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.130217 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.130231 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.130301 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:44.130452 (+ 151us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131287 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.132814 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133287 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133354 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133449 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.133543 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.133581 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":389,"thread_start_us":82,"threads_started":1}
I20260923 23:13:44.133656 2388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.129615 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:40867 (local address 127.0.0.1:34128)
0923 23:13:44.129917 (+ 302us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.129928 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.129978 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.130370 (+ 392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.130371 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.130373 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:44.130402 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.130403 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131358 (+ 955us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.131359 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.132744 (+ 1385us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.132745 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133361 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133362 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.133437 (+ 75us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133449 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.133510 (+ 61us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.133568 (+ 58us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":284,"mutex_wait_us":6,"thread_start_us":83,"threads_started":1}
I20260923 23:13:44.133824 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.130011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58630 (local address 127.16.122.130:34857)
0923 23:13:44.130200 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.130204 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.130212 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:44.130242 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.130244 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.130257 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.130334 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:44.130557 (+ 223us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131596 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.132565 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133294 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133368 (+ 74us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133460 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.133693 (+ 233us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.133733 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":159,"thread_start_us":88,"threads_started":1}
I20260923 23:13:44.133893 2391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.129806 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:34857 (local address 127.0.0.1:58630)
0923 23:13:44.130006 (+ 200us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.130027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.130063 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.130336 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.130336 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.130338 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:44.130370 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.130371 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131661 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.131662 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.132476 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.132478 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133488 (+ 1010us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133488 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.133553 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133559 (+ 6us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.133605 (+ 46us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.133651 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185,"thread_start_us":61,"threads_started":1}
I20260923 23:13:44.134007 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.129598 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:33101 (local address 127.0.0.1:43786)
0923 23:13:44.129721 (+ 123us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.129763 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.129824 (+ 61us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.130134 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.130137 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.130139 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:44.130207 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.130209 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131474 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.131475 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.132579 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.132582 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133486 (+ 904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133487 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.133553 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133561 (+ 8us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.133605 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.133653 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":96,"mutex_wait_us":10,"thread_start_us":71,"threads_started":1}
I20260923 23:13:44.134327 2051 tablet_service.cc:1467] Processing CreateTablet for tablet ffc2d5caa6894944a978d74bfe433280 (DEFAULT_TABLE table=test-table [id=6f8b271ea0ef48219eca87f56bfa4885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:44.134619 2051 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc2d5caa6894944a978d74bfe433280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:44.134593 2182 tablet_service.cc:1467] Processing CreateTablet for tablet ffc2d5caa6894944a978d74bfe433280 (DEFAULT_TABLE table=test-table [id=6f8b271ea0ef48219eca87f56bfa4885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:44.134893 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.129776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43786 (local address 127.16.122.131:33101)
0923 23:13:44.129966 (+ 190us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.129971 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.129981 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:13:44.130020 (+ 39us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.130022 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.130036 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.130118 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:44.130277 (+ 159us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.131400 (+ 1123us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.132645 (+ 1245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.133312 (+ 667us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.133368 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.133449 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.133693 (+ 244us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.134555 (+ 862us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158,"thread_start_us":89,"threads_started":1}
I20260923 23:13:44.134935 2182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc2d5caa6894944a978d74bfe433280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:44.135757 2313 tablet_service.cc:1467] Processing CreateTablet for tablet ffc2d5caa6894944a978d74bfe433280 (DEFAULT_TABLE table=test-table [id=6f8b271ea0ef48219eca87f56bfa4885]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:44.136085 2313 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ffc2d5caa6894944a978d74bfe433280. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:44.137194 2393 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Bootstrap starting.
I20260923 23:13:44.137840 2393 tablet_bootstrap.cc:654] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:44.138093 2394 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Bootstrap starting.
I20260923 23:13:44.138123 2393 log.cc:824] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:44.138638 2394 tablet_bootstrap.cc:654] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:44.138929 2393 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: No bootstrap required, opened a new log
I20260923 23:13:44.138911 2394 log.cc:824] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:44.138983 2393 ts_tablet_manager.cc:1397] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:44.139082 2396 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Bootstrap starting.
I20260923 23:13:44.139643 2396 tablet_bootstrap.cc:654] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:44.139900 2396 log.cc:824] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:44.140468 2396 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: No bootstrap required, opened a new log
I20260923 23:13:44.140519 2396 ts_tablet_manager.cc:1397] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:13:44.141177 2393 raft_consensus.cc:348] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.141309 2393 raft_consensus.cc:374] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:44.141338 2393 raft_consensus.cc:729] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eff68d4d2c2406c9a999afe173e918f, State: Initialized, Role: FOLLOWER
I20260923 23:13:44.141450 2393 consensus_queue.cc:260] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.141692 2393 ts_tablet_manager.cc:1428] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:44.141736 2117 heartbeater.cc:503] Master 127.16.122.190:41295 was elected leader, sending a full tablet report...
I20260923 23:13:44.142217 2394 tablet_bootstrap.cc:492] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: No bootstrap required, opened a new log
I20260923 23:13:44.142275 2394 ts_tablet_manager.cc:1397] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.001s
I20260923 23:13:44.142566 2396 raft_consensus.cc:348] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.142719 2396 raft_consensus.cc:374] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:44.142758 2396 raft_consensus.cc:729] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9270b204885b4219bd9e5e63712ea9bb, State: Initialized, Role: FOLLOWER
I20260923 23:13:44.142877 2396 consensus_queue.cc:260] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.143132 2396 ts_tablet_manager.cc:1428] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260923 23:13:44.143209 2379 heartbeater.cc:503] Master 127.16.122.190:41295 was elected leader, sending a full tablet report...
I20260923 23:13:44.144234 2394 raft_consensus.cc:348] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.144356 2394 raft_consensus.cc:374] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:44.144374 2394 raft_consensus.cc:729] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a69d06162043464cae35a23693f4d272, State: Initialized, Role: FOLLOWER
I20260923 23:13:44.144460 2394 consensus_queue.cc:260] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.144670 2394 ts_tablet_manager.cc:1428] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260923 23:13:44.144716 2248 heartbeater.cc:503] Master 127.16.122.190:41295 was elected leader, sending a full tablet report...
W20260923 23:13:44.221136 2249 tablet.cc:2369] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:13:44.356787 2118 tablet.cc:2369] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260923 23:13:44.361984 2380 tablet.cc:2369] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:13:44.384573 2399 raft_consensus.cc:482] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:44.384699 2399 raft_consensus.cc:504] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.385208 2399 leader_election.cc:288] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9270b204885b4219bd9e5e63712ea9bb (127.16.122.131:33101), a69d06162043464cae35a23693f4d272 (127.16.122.130:34857)
I20260923 23:13:44.388303 2392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.385677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:39885 (local address 127.16.122.130:34857)
0923 23:13:44.385733 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.385735 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.385837 (+ 102us) server_negotiation.cc:407] Connection header received
0923 23:13:44.385896 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.385897 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.385906 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.385959 (+ 53us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:44.386117 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.386583 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.387608 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.387968 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.388022 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.388030 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.388186 (+ 156us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.388223 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:44.388314 2402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.385413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:33101 (local address 127.16.122.129:55185)
0923 23:13:44.385547 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.385558 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.385626 (+ 68us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.385850 (+ 224us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.385851 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.385853 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:44.385930 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.385932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.386631 (+ 699us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.386632 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.387529 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.387532 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.388075 (+ 543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.388076 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.388134 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.388138 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.388186 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.388226 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":41,"thread_start_us":64,"threads_started":1}
I20260923 23:13:44.388311 2390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.385480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:55185 (local address 127.16.122.131:33101)
0923 23:13:44.385553 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.385556 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.385618 (+ 62us) server_negotiation.cc:407] Connection header received
0923 23:13:44.385726 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.385728 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.385738 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.385778 (+ 40us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:44.386015 (+ 237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.386547 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.387626 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.387986 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.388062 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.388071 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.388203 (+ 132us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.388240 (+ 37us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260923 23:13:44.388320 2403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.385604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:34857 (local address 127.16.122.129:39885)
0923 23:13:44.385710 (+ 106us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.385721 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:44.385842 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:44.385963 (+ 121us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:44.385964 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:44.385966 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:44.386051 (+ 85us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.386054 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.386667 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.386668 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.387514 (+ 846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:44.387516 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.388048 (+ 532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.388050 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:44.388117 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.388121 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:44.388170 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:13:44.388212 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90,"mutex_wait_us":2,"thread_start_us":60,"threads_started":1}
I20260923 23:13:44.388866 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc2d5caa6894944a978d74bfe433280" candidate_uuid: "5eff68d4d2c2406c9a999afe173e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9270b204885b4219bd9e5e63712ea9bb" is_pre_election: true
I20260923 23:13:44.388873 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc2d5caa6894944a978d74bfe433280" candidate_uuid: "5eff68d4d2c2406c9a999afe173e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d06162043464cae35a23693f4d272" is_pre_election: true
I20260923 23:13:44.389036 2333 raft_consensus.cc:2454] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eff68d4d2c2406c9a999afe173e918f in term 0.
I20260923 23:13:44.389036 2202 raft_consensus.cc:2454] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5eff68d4d2c2406c9a999afe173e918f in term 0.
I20260923 23:13:44.389272 2006 leader_election.cc:302] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eff68d4d2c2406c9a999afe173e918f, a69d06162043464cae35a23693f4d272; no voters:
I20260923 23:13:44.389415 2399 raft_consensus.cc:2781] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:44.389478 2399 raft_consensus.cc:482] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:44.389497 2399 raft_consensus.cc:3037] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:44.390246 2399 raft_consensus.cc:504] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.390374 2399 leader_election.cc:288] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [CANDIDATE]: Term 1 election: Requested vote from peers 9270b204885b4219bd9e5e63712ea9bb (127.16.122.131:33101), a69d06162043464cae35a23693f4d272 (127.16.122.130:34857)
I20260923 23:13:44.390597 2333 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc2d5caa6894944a978d74bfe433280" candidate_uuid: "5eff68d4d2c2406c9a999afe173e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9270b204885b4219bd9e5e63712ea9bb"
I20260923 23:13:44.390609 2202 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ffc2d5caa6894944a978d74bfe433280" candidate_uuid: "5eff68d4d2c2406c9a999afe173e918f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a69d06162043464cae35a23693f4d272"
I20260923 23:13:44.390683 2333 raft_consensus.cc:3037] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:44.390686 2202 raft_consensus.cc:3037] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:44.391480 2202 raft_consensus.cc:2454] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eff68d4d2c2406c9a999afe173e918f in term 1.
I20260923 23:13:44.391510 2333 raft_consensus.cc:2454] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5eff68d4d2c2406c9a999afe173e918f in term 1.
I20260923 23:13:44.391664 2006 leader_election.cc:302] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5eff68d4d2c2406c9a999afe173e918f, a69d06162043464cae35a23693f4d272; no voters:
I20260923 23:13:44.391762 2399 raft_consensus.cc:2781] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:44.391907 2399 raft_consensus.cc:686] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 1 LEADER]: Becoming Leader. State: Replica: 5eff68d4d2c2406c9a999afe173e918f, State: Running, Role: LEADER
I20260923 23:13:44.391988 2399 consensus_queue.cc:237] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } }
I20260923 23:13:44.392854 1929 catalog_manager.cc:5495] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f reported cstate change: term changed from 0 to 1, leader changed from <none> to 5eff68d4d2c2406c9a999afe173e918f (127.16.122.129). New cstate: current_term: 1 leader_uuid: "5eff68d4d2c2406c9a999afe173e918f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5eff68d4d2c2406c9a999afe173e918f" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 40867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 } health_report { overall_health: UNKNOWN } } }
I20260923 23:13:44.414023 2389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.412028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34134 (local address 127.16.122.129:40867)
0923 23:13:44.412083 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:44.412085 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:44.412094 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:44.412134 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:44.412134 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:44.412154 (+ 20us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:44.412212 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:44.412311 (+ 99us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.412779 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.413453 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:44.413631 (+ 178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:44.413674 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:44.413776 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:44.413874 (+ 98us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:44.413929 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:44.413945 (+ 16us) server_negotiation.cc:299] Negotiation successful
0923 23:13:44.413962 (+ 17us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:44.415804 2333 raft_consensus.cc:1261] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 1 FOLLOWER]: Refusing update from remote peer 5eff68d4d2c2406c9a999afe173e918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:44.415803 2202 raft_consensus.cc:1261] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 1 FOLLOWER]: Refusing update from remote peer 5eff68d4d2c2406c9a999afe173e918f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:44.416123 2404 consensus_queue.cc:1035] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "9270b204885b4219bd9e5e63712ea9bb" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 33101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:44.416252 2399 consensus_queue.cc:1035] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [LEADER]: Connected to new peer: Peer: permanent_uuid: "a69d06162043464cae35a23693f4d272" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 34857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:44.419023 2409 mvcc.cc:205] Tried to move back new op lower bound from 7332680599204614144 to 7332680599110340608. Current Snapshot: MvccSnapshot[applied={T|T < 7332680599204614144 or (T in {7332680599204614144})}]
I20260923 23:13:44.419369 2408 mvcc.cc:205] Tried to move back new op lower bound from 7332680599204614144 to 7332680599110340608. Current Snapshot: MvccSnapshot[applied={T|T < 7332680599204614144}]
I20260923 23:13:44.420557 1928 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:43014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:44.420647 1928 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:43014:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:44.421350 1928 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0232c4c778d643019f02127f880fb456: Sending DeleteTablet for 3 replicas of tablet ffc2d5caa6894944a978d74bfe433280
I20260923 23:13:44.421633 2313 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc2d5caa6894944a978d74bfe433280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:44 UTC) from {username='slave'} at 127.0.0.1:43786
I20260923 23:13:44.421679 2182 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc2d5caa6894944a978d74bfe433280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:44 UTC) from {username='slave'} at 127.0.0.1:58630
I20260923 23:13:44.421710 2051 tablet_service.cc:1514] Processing DeleteTablet for tablet ffc2d5caa6894944a978d74bfe433280 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:44 UTC) from {username='slave'} at 127.0.0.1:34128
I20260923 23:13:44.421885 2418 tablet_replica.cc:330] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: stopping tablet replica
I20260923 23:13:44.421976 2418 raft_consensus.cc:2229] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:44.422061 2418 raft_consensus.cc:2258] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:44.422163 2419 tablet_replica.cc:330] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f: stopping tablet replica
I20260923 23:13:44.422202 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1989
I20260923 23:13:44.422322 2419 raft_consensus.cc:2229] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:44.422473 2419 raft_consensus.cc:2258] T ffc2d5caa6894944a978d74bfe433280 P 5eff68d4d2c2406c9a999afe173e918f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:44.422514 2418 ts_tablet_manager.cc:1905] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:44.422775 2417 tablet_replica.cc:330] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: stopping tablet replica
I20260923 23:13:44.422897 2417 raft_consensus.cc:2229] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:44.422992 2417 raft_consensus.cc:2258] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:44.423405 2417 ts_tablet_manager.cc:1905] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:44.424268 2418 ts_tablet_manager.cc:1918] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:44.424341 2418 log.cc:1196] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/ffc2d5caa6894944a978d74bfe433280
I20260923 23:13:44.424573 2418 ts_tablet_manager.cc:1939] T ffc2d5caa6894944a978d74bfe433280 P a69d06162043464cae35a23693f4d272: Deleting consensus metadata
I20260923 23:13:44.424868 2417 ts_tablet_manager.cc:1918] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:44.424933 2417 log.cc:1196] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/ffc2d5caa6894944a978d74bfe433280
I20260923 23:13:44.424986 1915 catalog_manager.cc:4841] TS a69d06162043464cae35a23693f4d272 (127.16.122.130:34857): tablet ffc2d5caa6894944a978d74bfe433280 (table test-table [id=6f8b271ea0ef48219eca87f56bfa4885]) successfully deleted
I20260923 23:13:44.425143 2417 ts_tablet_manager.cc:1939] T ffc2d5caa6894944a978d74bfe433280 P 9270b204885b4219bd9e5e63712ea9bb: Deleting consensus metadata
I20260923 23:13:44.425561 1915 catalog_manager.cc:4841] TS 9270b204885b4219bd9e5e63712ea9bb (127.16.122.131:33101): tablet ffc2d5caa6894944a978d74bfe433280 (table test-table [id=6f8b271ea0ef48219eca87f56bfa4885]) successfully deleted
W20260923 23:13:44.428224 1914 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:40867 (error 108)
I20260923 23:13:44.428643 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2120
I20260923 23:13:44.428701 2387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:44.428557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:40867 (local address 127.0.0.1:34144)
0923 23:13:44.428623 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:44.428671 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:40867: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20260923 23:13:44.429317 1914 catalog_manager.cc:4584] TS 5eff68d4d2c2406c9a999afe173e918f (127.16.122.129:40867): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ffc2d5caa6894944a978d74bfe433280: Network error: Client connection negotiation failed: client connection to 127.16.122.129:40867: connect: Connection refused (error 111)
I20260923 23:13:44.433380 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2251
I20260923 23:13:44.437065 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 1898
2026-09-23T23:13:44Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (908 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:44 dist-test-slave-95hh krb5kdc[2423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:44 dist-test-slave-95hh krb5kdc[2423](info): set up 2 sockets
Sep 23 23:13:44 dist-test-slave-95hh krb5kdc[2423](info): commencing operation
krb5kdc: starting...
W20260923 23:13:45.374641 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:13:45.485857 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45663
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:45663
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:45.566385 2436 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:45.566670 2436 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:45.566717 2436 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:45.567911 2436 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:45.567950 2436 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:45.567970 2436 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:45.567984 2436 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:45.567999 2436 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:45.569446 2436 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:45663
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:45663
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2436
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:45.569692 2436 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:45.569896 2436 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:45.574438 2436 server_base.cc:1034] running on GCE node
W20260923 23:13:45.574395 2444 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:45.574390 2442 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:45.574492 2441 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:45.574821 2436 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:45.575137 2436 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:45.576275 2436 hybrid_clock.cc:648] HybridClock initialized: now 1790205225576263 us; error 30 us; skew 500 ppm
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:45.578560 2436 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:13:45.578953 2436 webserver.cc:460] Webserver started at http://127.16.122.190:40075/ using document root <none> and password file <none>
I20260923 23:13:45.579188 2436 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:45.579238 2436 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:45.579363 2436 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:45.580379 2436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "cfdfbbeeee97485b9b628b0ddedc6516"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "26f5da8bd8d69fe7a1b1bbb3ae771ca8"
server_key_iv: "20b0188af396440c86e2e39de462155e"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.580703 2436 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "cfdfbbeeee97485b9b628b0ddedc6516"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "26f5da8bd8d69fe7a1b1bbb3ae771ca8"
server_key_iv: "20b0188af396440c86e2e39de462155e"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.581885 2436 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:45.582508 2451 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.582654 2436 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:45.582727 2436 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "cfdfbbeeee97485b9b628b0ddedc6516"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "26f5da8bd8d69fe7a1b1bbb3ae771ca8"
server_key_iv: "20b0188af396440c86e2e39de462155e"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.582791 2436 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:45.599467 2436 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:45.601640 2436 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:45.601778 2436 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:45.605618 2436 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:45663
I20260923 23:13:45.605651 2503 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:45663 every 8 connection(s)
I20260923 23:13:45.605981 2436 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:45.606798 2504 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:45.609109 2504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: Bootstrap starting.
I20260923 23:13:45.609680 2504 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:45.609980 2504 log.cc:824] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:45.610081 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2436
I20260923 23:13:45.610195 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:45.610378 16874 external_mini_cluster.cc:1428] Setting key 0cdff0a1f2fcb5cd8b9b9199845d3682
I20260923 23:13:45.610747 2504 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: No bootstrap required, opened a new log
I20260923 23:13:45.612875 2504 raft_consensus.cc:348] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } }
I20260923 23:13:45.613015 2504 raft_consensus.cc:374] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:45.613045 2504 raft_consensus.cc:729] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cfdfbbeeee97485b9b628b0ddedc6516, State: Initialized, Role: FOLLOWER
I20260923 23:13:45.613168 2504 consensus_queue.cc:260] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } }
I20260923 23:13:45.613241 2504 raft_consensus.cc:388] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:45.613271 2504 raft_consensus.cc:482] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:45.613327 2504 raft_consensus.cc:3037] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:45.613848 2504 raft_consensus.cc:504] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } }
I20260923 23:13:45.613941 2504 leader_election.cc:302] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cfdfbbeeee97485b9b628b0ddedc6516; no voters:
I20260923 23:13:45.614095 2504 leader_election.cc:288] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:45.614130 2509 raft_consensus.cc:2781] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:45.614236 2509 raft_consensus.cc:686] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [term 1 LEADER]: Becoming Leader. State: Replica: cfdfbbeeee97485b9b628b0ddedc6516, State: Running, Role: LEADER
I20260923 23:13:45.614315 2509 consensus_queue.cc:237] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } }
I20260923 23:13:45.614356 2504 sys_catalog.cc:564] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:45.614620 2509 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cfdfbbeeee97485b9b628b0ddedc6516. Latest consensus state: current_term: 1 leader_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } } }
I20260923 23:13:45.614697 2509 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:45.614634 2510 sys_catalog.cc:455] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cfdfbbeeee97485b9b628b0ddedc6516" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 45663 } } }
I20260923 23:13:45.614861 2510 sys_catalog.cc:458] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516 [sys.catalog]: This master's current role is: LEADER
W20260923 23:13:45.616127 2524 catalog_manager.cc:1558] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:13:45.616178 2524 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:13:45.616215 2514 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:45.616344 2514 catalog_manager.cc:1484] Initializing Kudu cluster ID...
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:45.618546 2514 catalog_manager.cc:1347] Generated new cluster ID: f8b7997cf6104000a92a4be4adea3177
I20260923 23:13:45.618598 2514 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:45.621556 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:45.610872 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56682 (local address 127.16.122.190:45663)
0923 23:13:45.611070 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:45.611080 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:45.611113 (+ 33us) server_negotiation.cc:407] Connection header received
0923 23:13:45.611965 (+ 852us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:45.611972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:45.612023 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:45.612141 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:45.612878 (+ 737us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.614108 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.615167 (+ 1059us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.615685 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.616080 (+ 395us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:45.618369 (+ 2289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:45.618373 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:45.618373 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:45.618386 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:45.620091 (+ 1705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.620411 (+ 320us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.620414 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.620414 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.620505 (+ 91us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.620707 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.620708 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.620709 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.620962 (+ 253us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:45.621116 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:45.621211 (+ 95us) server_negotiation.cc:299] Negotiation successful
0923 23:13:45.621355 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":72,"threads_started":1}
I20260923 23:13:45.627583 2514 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:45.628111 2514 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:45.634712 2514 catalog_manager.cc:5868] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: Generated new TSK 0
I20260923 23:13:45.634927 2514 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:13:45.667929 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:45.748255 2530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:45.748474 2530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:45.748498 2530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:45.749670 2530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:45.749718 2530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:45.749747 2530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:45.751264 2530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2530
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:45.751505 2530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:45.751726 2530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:45.756659 2536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:45.756634 2538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:45.756671 2530 server_base.cc:1034] running on GCE node
W20260923 23:13:45.756629 2535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:45.757016 2530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:45.757269 2530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:45.758419 2530 hybrid_clock.cc:648] HybridClock initialized: now 1790205225758404 us; error 32 us; skew 500 ppm
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:45.760834 2530 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:13:45.761242 2530 webserver.cc:460] Webserver started at http://127.16.122.129:39031/ using document root <none> and password file <none>
I20260923 23:13:45.761445 2530 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:45.761489 2530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:45.761615 2530 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:45.762665 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "ae62454989504c599a9d0493f63dd8d1"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "018b21d2523332d386cf30d838982f78"
server_key_iv: "b21c45aae88e5a90bbaecc259eb5c83a"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.762990 2530 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "ae62454989504c599a9d0493f63dd8d1"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "018b21d2523332d386cf30d838982f78"
server_key_iv: "b21c45aae88e5a90bbaecc259eb5c83a"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.764209 2530 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:45.764839 2545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.764973 2530 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:45.765048 2530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "ae62454989504c599a9d0493f63dd8d1"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "018b21d2523332d386cf30d838982f78"
server_key_iv: "b21c45aae88e5a90bbaecc259eb5c83a"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.765113 2530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:45.788755 2530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:45.790894 2530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:45.791026 2530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:45.791309 2530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:45.791611 2530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:45.791644 2530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.791666 2530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:45.791689 2530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.797071 2530 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:42911
I20260923 23:13:45.797113 2658 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:42911 every 8 connection(s)
I20260923 23:13:45.797418 2530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:45.802649 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2530
I20260923 23:13:45.802739 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:45.802919 16874 external_mini_cluster.cc:1428] Setting key 2ba10bf8781918f9ace51af212b20552
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:45.805991 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:45.798079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:53659 (local address 127.16.122.190:45663)
0923 23:13:45.798135 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:45.798137 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:45.798330 (+ 193us) server_negotiation.cc:407] Connection header received
0923 23:13:45.798792 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:45.798793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:45.798801 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:45.798857 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:45.799773 (+ 916us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.800271 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.801188 (+ 917us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.801390 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.801436 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:45.803464 (+ 2028us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:45.803465 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:45.803466 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:45.803473 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:45.804745 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.805168 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.805168 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.805169 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.805212 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.805437 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.805438 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.805438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.805530 (+ 92us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:45.805572 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:45.805865 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 23:13:45.805928 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:13:45.806213 2661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:45.798054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45663 (local address 127.16.122.129:53659)
0923 23:13:45.798202 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:45.798243 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:45.798718 (+ 475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:45.798911 (+ 193us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:45.798914 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:45.799495 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:45.799694 (+ 199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:45.799700 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.800344 (+ 644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.800345 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:45.801127 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:45.801129 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.801445 (+ 316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.801446 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:45.801544 (+ 98us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:45.801988 (+ 444us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:45.801996 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:45.803382 (+ 1386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:45.804820 (+ 1438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:45.804827 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:45.804829 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:45.805102 (+ 273us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:45.805251 (+ 149us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:45.805252 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:45.805252 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:45.805384 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:45.805581 (+ 197us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:45.805586 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:45.805733 (+ 147us) client_negotiation.cc:769] Sending connection context
0923 23:13:45.805846 (+ 113us) client_negotiation.cc:240] Negotiation successful
0923 23:13:45.805998 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105,"thread_start_us":68,"threads_started":1}
I20260923 23:13:45.806730 2659 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45663
I20260923 23:13:45.806838 2659 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:45.807008 2659 heartbeater.cc:511] Master 127.16.122.190:45663 requested a full tablet report, sending...
I20260923 23:13:45.807596 2468 ts_manager.cc:194] Registered new tserver with Master: ae62454989504c599a9d0493f63dd8d1 (127.16.122.129:42911)
I20260923 23:13:45.808475 2468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:53659
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:13:45.825662 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:45.905241 2664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:45.905479 2664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:45.905503 2664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:45.906634 2664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:45.906673 2664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:45.906699 2664 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:45.908175 2664 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2664
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:45.908397 2664 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:45.908607 2664 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:45.913177 2672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:45.913177 2669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:45.913184 2670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:45.913376 2664 server_base.cc:1034] running on GCE node
I20260923 23:13:45.913552 2664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:45.913787 2664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:45.914945 2664 hybrid_clock.cc:648] HybridClock initialized: now 1790205225914931 us; error 29 us; skew 500 ppm
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:45.917331 2664 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:13:45.917740 2664 webserver.cc:460] Webserver started at http://127.16.122.130:45283/ using document root <none> and password file <none>
I20260923 23:13:45.917930 2664 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:45.917973 2664 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:45.918097 2664 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:45.919189 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "5c599e6af1164bbaa653e57735db0dc9"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "692fdc8fca0f1b2ab1b3101b36a6b464"
server_key_iv: "a4428418c961ac3815f98d92e80c7024"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.919524 2664 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "5c599e6af1164bbaa653e57735db0dc9"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "692fdc8fca0f1b2ab1b3101b36a6b464"
server_key_iv: "a4428418c961ac3815f98d92e80c7024"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.920745 2664 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:45.921408 2679 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.921552 2664 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:45.921629 2664 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "5c599e6af1164bbaa653e57735db0dc9"
format_stamp: "Formatted at 2026-09-23 23:13:45 on dist-test-slave-95hh"
server_key: "692fdc8fca0f1b2ab1b3101b36a6b464"
server_key_iv: "a4428418c961ac3815f98d92e80c7024"
server_key_version: "encryptionkey@0"
I20260923 23:13:45.921697 2664 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:45.937348 2664 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:45.939553 2664 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:45.939685 2664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:45.939906 2664 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:45.940194 2664 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:45.940227 2664 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.940248 2664 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:45.940279 2664 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:45.945662 2664 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:37317
I20260923 23:13:45.945715 2792 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:37317 every 8 connection(s)
I20260923 23:13:45.946017 2664 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:45.950069 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2664
I20260923 23:13:45.950188 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:45.950385 16874 external_mini_cluster.cc:1428] Setting key 4305f6a5e02531009b993a311c8c9e4e
Sep 23 23:13:45 dist-test-slave-95hh krb5kdc[2423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205225, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:45.955123 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:45.946661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:41973 (local address 127.16.122.190:45663)
0923 23:13:45.946704 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:45.946705 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:45.946896 (+ 191us) server_negotiation.cc:407] Connection header received
0923 23:13:45.947440 (+ 544us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:45.947442 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:45.947454 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:45.947526 (+ 72us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:45.948428 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.948918 (+ 490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.949847 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.950041 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.950084 (+ 43us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:45.952502 (+ 2418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:45.952504 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:45.952504 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:45.952512 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:45.953793 (+ 1281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.954160 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.954161 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.954162 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.954222 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:45.954452 (+ 230us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:45.954453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:45.954453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:45.954570 (+ 117us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:45.954626 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:45.954906 (+ 280us) server_negotiation.cc:299] Negotiation successful
0923 23:13:45.954995 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:13:45.955312 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:45.946638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45663 (local address 127.16.122.130:41973)
0923 23:13:45.946778 (+ 140us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:45.946815 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:45.947321 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:45.947579 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:45.947582 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:45.948134 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:45.948339 (+ 205us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:45.948345 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.948984 (+ 639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.948985 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:45.949788 (+ 803us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:45.949790 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:45.950110 (+ 320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:45.950111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:45.950289 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:45.950814 (+ 525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:45.950824 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:45.952408 (+ 1584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:45.953871 (+ 1463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:45.953878 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:45.953880 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:45.954093 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:45.954281 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:45.954282 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:45.954283 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:45.954396 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:45.954626 (+ 230us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:45.954633 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:45.954785 (+ 152us) client_negotiation.cc:769] Sending connection context
0923 23:13:45.954877 (+ 92us) client_negotiation.cc:240] Negotiation successful
0923 23:13:45.955067 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"thread_start_us":88,"threads_started":1}
I20260923 23:13:45.955969 2793 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45663
I20260923 23:13:45.956058 2793 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:45.956229 2793 heartbeater.cc:511] Master 127.16.122.190:45663 requested a full tablet report, sending...
I20260923 23:13:45.956602 2468 ts_manager.cc:194] Registered new tserver with Master: 5c599e6af1164bbaa653e57735db0dc9 (127.16.122.130:37317)
I20260923 23:13:45.956974 2468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:41973
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:13:45.974095 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:46853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:46.053258 2798 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:46.053467 2798 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:46.053488 2798 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:46.054579 2798 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:46.054610 2798 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:46.054634 2798 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:46.056070 2798 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:46853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:45663
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2798
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:46.056288 2798 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:46.056468 2798 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:46.061220 2804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.061283 2798 server_base.cc:1034] running on GCE node
W20260923 23:13:46.061241 2803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.061232 2806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.061594 2798 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:46.061828 2798 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:46.062964 2798 hybrid_clock.cc:648] HybridClock initialized: now 1790205226062954 us; error 29 us; skew 500 ppm
Sep 23 23:13:46 dist-test-slave-95hh krb5kdc[2423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:46.065218 2798 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:13:46.065634 2798 webserver.cc:460] Webserver started at http://127.16.122.131:45701/ using document root <none> and password file <none>
I20260923 23:13:46.065826 2798 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:46.065866 2798 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:46.065953 2798 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:46.066926 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "572647c78beb4603a5497dab0e96136a"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "b5af90a8655bab156cbc80105a797aa0"
server_key_iv: "2633734c21b6feb747b76f12ca6cecba"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.067255 2798 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "572647c78beb4603a5497dab0e96136a"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "b5af90a8655bab156cbc80105a797aa0"
server_key_iv: "2633734c21b6feb747b76f12ca6cecba"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.068476 2798 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:46.069135 2813 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.069284 2798 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:46.069365 2798 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "572647c78beb4603a5497dab0e96136a"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "b5af90a8655bab156cbc80105a797aa0"
server_key_iv: "2633734c21b6feb747b76f12ca6cecba"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.069419 2798 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:46.094056 2798 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:46.096227 2798 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:46.096349 2798 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:46.096532 2798 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:46.096788 2798 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:46.096817 2798 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.096838 2798 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:46.096853 2798 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.102113 2798 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:37743
I20260923 23:13:46.102178 2926 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:37743 every 8 connection(s)
I20260923 23:13:46.102456 2798 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
Sep 23 23:13:46 dist-test-slave-95hh krb5kdc[2423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205226, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:46.108606 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2798
I20260923 23:13:46.108716 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:46.108896 16874 external_mini_cluster.cc:1428] Setting key 9f85ba824f71813f4696aa3a7053508a
I20260923 23:13:46.111297 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.103148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:60047 (local address 127.16.122.190:45663)
0923 23:13:46.103188 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.103190 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.103407 (+ 217us) server_negotiation.cc:407] Connection header received
0923 23:13:46.103916 (+ 509us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.103917 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.103931 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.103981 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.104903 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.105416 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.106395 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.106586 (+ 191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.106642 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.108786 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.108788 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.108788 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:46.108796 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.110137 (+ 1341us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:46.110473 (+ 336us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:46.110474 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:46.110474 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:46.110535 (+ 61us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:46.110767 (+ 232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:46.110767 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:46.110767 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:46.110860 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.110905 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.111158 (+ 253us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.111227 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:13:46.111464 2929 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.103144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:45663 (local address 127.16.122.131:60047)
0923 23:13:46.103292 (+ 148us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.103327 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.103820 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.104053 (+ 233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.104057 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.104602 (+ 545us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:46.104808 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.104814 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.105501 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.105502 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.106311 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.106314 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.106666 (+ 352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.106667 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.106800 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.107304 (+ 504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:46.107312 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:46.108682 (+ 1370us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:46.110204 (+ 1522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:46.110212 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:46.110213 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:46.110417 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:46.110598 (+ 181us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:46.110599 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:46.110599 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:46.110719 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:46.110904 (+ 185us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:46.110907 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:46.111021 (+ 114us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.111145 (+ 124us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.111274 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260923 23:13:46.111857 2927 heartbeater.cc:347] Connected to a master server at 127.16.122.190:45663
I20260923 23:13:46.111948 2927 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:46.112118 2927 heartbeater.cc:511] Master 127.16.122.190:45663 requested a full tablet report, sending...
I20260923 23:13:46.112438 2468 ts_manager.cc:194] Registered new tserver with Master: 572647c78beb4603a5497dab0e96136a (127.16.122.131:37743)
I20260923 23:13:46.112798 2468 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:60047
I20260923 23:13:46.120667 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:46.127753 2508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.121992 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56694 (local address 127.16.122.190:45663)
0923 23:13:46.122047 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.122049 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.122125 (+ 76us) server_negotiation.cc:407] Connection header received
0923 23:13:46.122175 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.122176 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.122184 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.122235 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.122754 (+ 519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.123294 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.123995 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.124170 (+ 175us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.124211 (+ 41us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.125491 (+ 1280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.125492 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.125492 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:46.125499 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.126749 (+ 1250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:46.127115 (+ 366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:46.127116 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:46.127116 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:46.127167 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:46.127321 (+ 154us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:46.127321 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:46.127322 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:46.127447 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.127522 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.127618 (+ 96us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.127664 (+ 46us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:13:46.129525 2468 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56694:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:46.131199 2468 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:46.137404 2935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:37317 (local address 127.0.0.1:44866)
0923 23:13:46.133643 (+ 298us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.133689 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.133764 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.133976 (+ 212us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.133980 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.133984 (+ 4us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.134085 (+ 101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.134087 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.135121 (+ 1034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.135121 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.136314 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.136317 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137107 (+ 790us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137108 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.137185 (+ 77us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137198 (+ 13us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.137263 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.137320 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":266,"thread_start_us":64,"threads_started":1}
I20260923 23:13:46.137607 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48258 (local address 127.16.122.131:37743)
0923 23:13:46.133662 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.133668 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.134091 (+ 423us) server_negotiation.cc:407] Connection header received
0923 23:13:46.134148 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.134150 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.134164 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.134230 (+ 66us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.134353 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.135475 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.136452 (+ 977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137084 (+ 632us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137151 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137205 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.137465 (+ 260us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.137510 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153,"thread_start_us":82,"threads_started":1}
I20260923 23:13:46.137717 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44866 (local address 127.16.122.130:37317)
0923 23:13:46.133747 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.133751 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.133761 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:13:46.133856 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.133859 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.133874 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.133954 (+ 80us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.134189 (+ 235us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.134989 (+ 800us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.136408 (+ 1419us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137023 (+ 615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137098 (+ 75us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137165 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.137600 (+ 435us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.137643 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":277,"thread_start_us":100,"threads_started":1}
I20260923 23:13:46.137825 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:37743 (local address 127.0.0.1:48258)
0923 23:13:46.133799 (+ 454us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.134028 (+ 229us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.134074 (+ 46us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.134227 (+ 153us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.134228 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.134230 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.134276 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.134277 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.135630 (+ 1353us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.135639 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.136377 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.136379 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137236 (+ 857us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137236 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.137374 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137377 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.137716 (+ 339us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.137760 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":443,"mutex_wait_us":2,"thread_start_us":63,"threads_started":1}
I20260923 23:13:46.138132 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133670 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49088 (local address 127.16.122.129:42911)
0923 23:13:46.133840 (+ 170us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.133844 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.133917 (+ 73us) server_negotiation.cc:407] Connection header received
0923 23:13:46.133990 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.133992 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.134008 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.134092 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.134403 (+ 311us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.135520 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.136805 (+ 1285us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137508 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137572 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137621 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.138011 (+ 390us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.138054 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":139,"thread_start_us":90,"threads_started":1}
I20260923 23:13:46.138357 2936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.133549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42911 (local address 127.0.0.1:49088)
0923 23:13:46.133871 (+ 322us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.133878 (+ 7us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.133926 (+ 48us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.134299 (+ 373us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.134300 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.134301 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.134332 (+ 31us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.134333 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.135619 (+ 1286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.135620 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.136723 (+ 1103us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.136725 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.137663 (+ 938us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.137664 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.137721 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.137724 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.138240 (+ 516us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.138287 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":313,"mutex_wait_us":25,"thread_start_us":49,"threads_started":1}
I20260923 23:13:46.138437 2861 tablet_service.cc:1467] Processing CreateTablet for tablet d3936cdea1374f42bbfc49662251736d (DEFAULT_TABLE table=test-table [id=9fa4fc0d91de43aeb49f4c09b56a8f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.138437 2727 tablet_service.cc:1467] Processing CreateTablet for tablet d3936cdea1374f42bbfc49662251736d (DEFAULT_TABLE table=test-table [id=9fa4fc0d91de43aeb49f4c09b56a8f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.138799 2861 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3936cdea1374f42bbfc49662251736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.138799 2727 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3936cdea1374f42bbfc49662251736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.138942 2593 tablet_service.cc:1467] Processing CreateTablet for tablet d3936cdea1374f42bbfc49662251736d (DEFAULT_TABLE table=test-table [id=9fa4fc0d91de43aeb49f4c09b56a8f18]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.139292 2593 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet d3936cdea1374f42bbfc49662251736d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.141912 2941 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: Bootstrap starting.
I20260923 23:13:46.142076 2942 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Bootstrap starting.
I20260923 23:13:46.142215 2943 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Bootstrap starting.
I20260923 23:13:46.142727 2943 tablet_bootstrap.cc:654] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.142735 2942 tablet_bootstrap.cc:654] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.142735 2941 tablet_bootstrap.cc:654] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.143291 2943 log.cc:824] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.143303 2942 log.cc:824] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.143501 2941 log.cc:824] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.144024 2942 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: No bootstrap required, opened a new log
I20260923 23:13:46.144081 2942 ts_tablet_manager.cc:1397] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:46.144205 2943 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: No bootstrap required, opened a new log
I20260923 23:13:46.144263 2943 ts_tablet_manager.cc:1397] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:13:46.145486 2942 raft_consensus.cc:348] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.145625 2942 raft_consensus.cc:374] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.145649 2942 raft_consensus.cc:729] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5c599e6af1164bbaa653e57735db0dc9, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.145581 2943 raft_consensus.cc:348] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.145740 2943 raft_consensus.cc:374] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.145782 2943 raft_consensus.cc:729] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 572647c78beb4603a5497dab0e96136a, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.145753 2942 consensus_queue.cc:260] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.145890 2943 consensus_queue.cc:260] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.146010 2793 heartbeater.cc:503] Master 127.16.122.190:45663 was elected leader, sending a full tablet report...
I20260923 23:13:46.146142 2943 ts_tablet_manager.cc:1428] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:13:46.146145 2927 heartbeater.cc:503] Master 127.16.122.190:45663 was elected leader, sending a full tablet report...
I20260923 23:13:46.146347 2942 ts_tablet_manager.cc:1428] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:46.147023 2941 tablet_bootstrap.cc:492] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: No bootstrap required, opened a new log
I20260923 23:13:46.147109 2941 ts_tablet_manager.cc:1397] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: Time spent bootstrapping tablet: real 0.005s user 0.001s sys 0.001s
I20260923 23:13:46.148558 2941 raft_consensus.cc:348] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.148700 2941 raft_consensus.cc:374] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.148722 2941 raft_consensus.cc:729] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae62454989504c599a9d0493f63dd8d1, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.148815 2941 consensus_queue.cc:260] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.149058 2941 ts_tablet_manager.cc:1428] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260923 23:13:46.149115 2659 heartbeater.cc:503] Master 127.16.122.190:45663 was elected leader, sending a full tablet report...
I20260923 23:13:46.166877 2947 raft_consensus.cc:482] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:46.166996 2947 raft_consensus.cc:504] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.167414 2947 leader_election.cc:288] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ae62454989504c599a9d0493f63dd8d1 (127.16.122.129:42911), 572647c78beb4603a5497dab0e96136a (127.16.122.131:37743)
I20260923 23:13:46.171094 2938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.167745 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:51485 (local address 127.16.122.131:37743)
0923 23:13:46.167818 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.167820 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.167828 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:46.167853 (+ 25us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.167854 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.167864 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.167905 (+ 41us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.168144 (+ 239us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.168782 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170160 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.170738 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170807 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.170816 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.170959 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.170999 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20260923 23:13:46.171094 2940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.167809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:59027 (local address 127.16.122.129:42911)
0923 23:13:46.167856 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.167858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.168023 (+ 165us) server_negotiation.cc:407] Connection header received
0923 23:13:46.168120 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.168121 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.168134 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.168194 (+ 60us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.168325 (+ 131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.168984 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170218 (+ 1234us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.170740 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170787 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.170796 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.170969 (+ 173us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.171008 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:13:46.171118 2795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.167596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:37743 (local address 127.16.122.130:51485)
0923 23:13:46.167660 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.167672 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.167719 (+ 47us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.167966 (+ 247us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.167967 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.167969 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.168038 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.168041 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.168857 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.168858 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.170061 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.170064 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.170808 (+ 744us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170809 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.170891 (+ 82us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.170895 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.170950 (+ 55us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.170989 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":43,"mutex_wait_us":4}
I20260923 23:13:46.171123 2950 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.167600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42911 (local address 127.16.122.130:59027)
0923 23:13:46.167968 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.167977 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.168044 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.168200 (+ 156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.168201 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.168202 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.168259 (+ 57us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.168260 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.169054 (+ 794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.169055 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.170128 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.170130 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.170847 (+ 717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.170847 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.170907 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.170909 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.170954 (+ 45us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.170988 (+ 34us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":351,"mutex_wait_us":55,"thread_start_us":130,"threads_started":1}
I20260923 23:13:46.171474 2881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3936cdea1374f42bbfc49662251736d" candidate_uuid: "5c599e6af1164bbaa653e57735db0dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572647c78beb4603a5497dab0e96136a" is_pre_election: true
I20260923 23:13:46.171530 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3936cdea1374f42bbfc49662251736d" candidate_uuid: "5c599e6af1164bbaa653e57735db0dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62454989504c599a9d0493f63dd8d1" is_pre_election: true
I20260923 23:13:46.171656 2613 raft_consensus.cc:2454] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c599e6af1164bbaa653e57735db0dc9 in term 0.
I20260923 23:13:46.171656 2881 raft_consensus.cc:2454] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5c599e6af1164bbaa653e57735db0dc9 in term 0.
I20260923 23:13:46.171857 2680 leader_election.cc:302] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572647c78beb4603a5497dab0e96136a, 5c599e6af1164bbaa653e57735db0dc9; no voters:
I20260923 23:13:46.171973 2947 raft_consensus.cc:2781] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:46.172027 2947 raft_consensus.cc:482] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:46.172055 2947 raft_consensus.cc:3037] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.172736 2947 raft_consensus.cc:504] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.172875 2947 leader_election.cc:288] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [CANDIDATE]: Term 1 election: Requested vote from peers ae62454989504c599a9d0493f63dd8d1 (127.16.122.129:42911), 572647c78beb4603a5497dab0e96136a (127.16.122.131:37743)
I20260923 23:13:46.173031 2881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3936cdea1374f42bbfc49662251736d" candidate_uuid: "5c599e6af1164bbaa653e57735db0dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "572647c78beb4603a5497dab0e96136a"
I20260923 23:13:46.173115 2881 raft_consensus.cc:3037] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.173130 2613 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "d3936cdea1374f42bbfc49662251736d" candidate_uuid: "5c599e6af1164bbaa653e57735db0dc9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae62454989504c599a9d0493f63dd8d1"
I20260923 23:13:46.173200 2613 raft_consensus.cc:3037] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.173969 2881 raft_consensus.cc:2454] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c599e6af1164bbaa653e57735db0dc9 in term 1.
I20260923 23:13:46.173971 2613 raft_consensus.cc:2454] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5c599e6af1164bbaa653e57735db0dc9 in term 1.
I20260923 23:13:46.174119 2680 leader_election.cc:302] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 572647c78beb4603a5497dab0e96136a, 5c599e6af1164bbaa653e57735db0dc9; no voters:
I20260923 23:13:46.174218 2947 raft_consensus.cc:2781] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:46.174369 2947 raft_consensus.cc:686] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 1 LEADER]: Becoming Leader. State: Replica: 5c599e6af1164bbaa653e57735db0dc9, State: Running, Role: LEADER
I20260923 23:13:46.174441 2947 consensus_queue.cc:237] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } }
I20260923 23:13:46.175264 2466 catalog_manager.cc:5495] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5c599e6af1164bbaa653e57735db0dc9 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "5c599e6af1164bbaa653e57735db0dc9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5c599e6af1164bbaa653e57735db0dc9" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 37317 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 } health_report { overall_health: UNKNOWN } } }
I20260923 23:13:46.184060 2937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.182037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44868 (local address 127.16.122.130:37317)
0923 23:13:46.182075 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.182077 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.182088 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:13:46.182155 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.182156 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.182171 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.182229 (+ 58us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:46.182321 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.182772 (+ 451us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.183468 (+ 696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.183642 (+ 174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.183688 (+ 46us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.183790 (+ 102us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:46.183874 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:46.183939 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.183958 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.183967 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:13:46.185827 2881 raft_consensus.cc:1261] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 1 FOLLOWER]: Refusing update from remote peer 5c599e6af1164bbaa653e57735db0dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:46.185828 2613 raft_consensus.cc:1261] T d3936cdea1374f42bbfc49662251736d P ae62454989504c599a9d0493f63dd8d1 [term 1 FOLLOWER]: Refusing update from remote peer 5c599e6af1164bbaa653e57735db0dc9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:46.186203 2951 consensus_queue.cc:1035] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "572647c78beb4603a5497dab0e96136a" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 37743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:46.186331 2947 consensus_queue.cc:1035] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae62454989504c599a9d0493f63dd8d1" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 42911 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:46.189347 2955 mvcc.cc:205] Tried to move back new op lower bound from 7332680606454673408 to 7332680606411157504. Current Snapshot: MvccSnapshot[applied={T|T < 7332680606454673408 or (T in {7332680606454673408})}]
I20260923 23:13:46.189556 2956 mvcc.cc:205] Tried to move back new op lower bound from 7332680606454673408 to 7332680606411157504. Current Snapshot: MvccSnapshot[applied={T|T < 7332680606454673408}]
I20260923 23:13:46.190852 2466 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:46.190951 2466 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:56694:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:46.191707 2466 catalog_manager.cc:5782] T 00000000000000000000000000000000 P cfdfbbeeee97485b9b628b0ddedc6516: Sending DeleteTablet for 3 replicas of tablet d3936cdea1374f42bbfc49662251736d
I20260923 23:13:46.192054 2861 tablet_service.cc:1514] Processing DeleteTablet for tablet d3936cdea1374f42bbfc49662251736d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:48258
I20260923 23:13:46.192104 2593 tablet_service.cc:1514] Processing DeleteTablet for tablet d3936cdea1374f42bbfc49662251736d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:49088
I20260923 23:13:46.192128 2727 tablet_service.cc:1514] Processing DeleteTablet for tablet d3936cdea1374f42bbfc49662251736d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:44866
I20260923 23:13:46.192286 2963 tablet_replica.cc:330] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: stopping tablet replica
I20260923 23:13:46.192330 2965 tablet_replica.cc:330] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: stopping tablet replica
I20260923 23:13:46.192378 2963 raft_consensus.cc:2229] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:46.192410 2965 raft_consensus.cc:2229] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:46.192479 2963 raft_consensus.cc:2258] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:46.192515 2965 raft_consensus.cc:2258] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:46.192765 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2530
I20260923 23:13:46.192917 2963 ts_tablet_manager.cc:1905] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:46.192943 2965 ts_tablet_manager.cc:1905] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:46.194590 2965 ts_tablet_manager.cc:1918] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:46.194597 2963 ts_tablet_manager.cc:1918] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:46.194653 2965 log.cc:1196] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/d3936cdea1374f42bbfc49662251736d
I20260923 23:13:46.194654 2963 log.cc:1196] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/d3936cdea1374f42bbfc49662251736d
I20260923 23:13:46.194911 2963 ts_tablet_manager.cc:1939] T d3936cdea1374f42bbfc49662251736d P 572647c78beb4603a5497dab0e96136a: Deleting consensus metadata
I20260923 23:13:46.194933 2965 ts_tablet_manager.cc:1939] T d3936cdea1374f42bbfc49662251736d P 5c599e6af1164bbaa653e57735db0dc9: Deleting consensus metadata
I20260923 23:13:46.195312 2452 catalog_manager.cc:4841] TS 572647c78beb4603a5497dab0e96136a (127.16.122.131:37743): tablet d3936cdea1374f42bbfc49662251736d (table test-table [id=9fa4fc0d91de43aeb49f4c09b56a8f18]) successfully deleted
I20260923 23:13:46.195327 2453 catalog_manager.cc:4841] TS 5c599e6af1164bbaa653e57735db0dc9 (127.16.122.130:37317): tablet d3936cdea1374f42bbfc49662251736d (table test-table [id=9fa4fc0d91de43aeb49f4c09b56a8f18]) successfully deleted
W20260923 23:13:46.199054 2453 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:42911 (error 108)
I20260923 23:13:46.199280 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2664
I20260923 23:13:46.200804 2939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.199901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:42911 (local address 127.0.0.1:49090)
0923 23:13:46.200734 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.200777 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42911: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":811}
W20260923 23:13:46.200929 2453 catalog_manager.cc:4584] TS ae62454989504c599a9d0493f63dd8d1 (127.16.122.129:42911): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet d3936cdea1374f42bbfc49662251736d: Network error: Client connection negotiation failed: client connection to 127.16.122.129:42911: connect: Connection refused (error 111)
I20260923 23:13:46.204211 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2798
I20260923 23:13:46.208307 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2436
2026-09-23T23:13:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (1772 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-09-23T23:13:46Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:46Z Disabled control of system clock
I20260923 23:13:46.233623 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:46173
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:46173
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:46.311360 2970 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:46.311578 2970 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:46.311595 2970 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:46.312760 2970 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:46.312803 2970 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:46.312813 2970 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:46.312835 2970 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:46.312860 2970 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:46.314189 2970 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:46173
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:46173
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.2970
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:46.314431 2970 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:46.314707 2970 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:46.319327 2975 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.319339 2978 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.319339 2976 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.319368 2970 server_base.cc:1034] running on GCE node
I20260923 23:13:46.319733 2970 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:46.319996 2970 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:46.321132 2970 hybrid_clock.cc:648] HybridClock initialized: now 1790205226321119 us; error 31 us; skew 500 ppm
I20260923 23:13:46.322317 2970 webserver.cc:460] Webserver started at http://127.16.122.190:42773/ using document root <none> and password file <none>
I20260923 23:13:46.322531 2970 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:46.322580 2970 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:46.322705 2970 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:46.323789 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "609a950c305e494d893c585e8fe379cc"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "aa4cd107af35b8985c5bfcce8fbd1cea"
server_key_iv: "5fa3b4fd2261a2db789979bf0234d6cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.324112 2970 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "609a950c305e494d893c585e8fe379cc"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "aa4cd107af35b8985c5bfcce8fbd1cea"
server_key_iv: "5fa3b4fd2261a2db789979bf0234d6cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.325291 2970 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:46.325932 2984 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.326094 2970 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:46.326181 2970 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "609a950c305e494d893c585e8fe379cc"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "aa4cd107af35b8985c5bfcce8fbd1cea"
server_key_iv: "5fa3b4fd2261a2db789979bf0234d6cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.326259 2970 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:46.345120 2970 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:46.345469 2970 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:46.345603 2970 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:46.349279 2970 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:46173
I20260923 23:13:46.349313 3036 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:46173 every 8 connection(s)
I20260923 23:13:46.349635 2970 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:46.350531 3037 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.352839 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: Bootstrap starting.
I20260923 23:13:46.353359 3037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.353590 3037 log.cc:824] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.354183 3037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: No bootstrap required, opened a new log
I20260923 23:13:46.355783 3037 raft_consensus.cc:348] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } }
I20260923 23:13:46.355924 3037 raft_consensus.cc:374] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.355952 3037 raft_consensus.cc:729] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 609a950c305e494d893c585e8fe379cc, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.356058 3037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } }
I20260923 23:13:46.356107 3037 raft_consensus.cc:388] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:46.356125 3037 raft_consensus.cc:482] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:46.356155 3037 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.356595 3037 raft_consensus.cc:504] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } }
I20260923 23:13:46.356683 3037 leader_election.cc:302] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 609a950c305e494d893c585e8fe379cc; no voters:
I20260923 23:13:46.356813 3037 leader_election.cc:288] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:46.356858 3040 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:46.357002 3040 raft_consensus.cc:686] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [term 1 LEADER]: Becoming Leader. State: Replica: 609a950c305e494d893c585e8fe379cc, State: Running, Role: LEADER
I20260923 23:13:46.357051 3037 sys_catalog.cc:564] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:46.357102 3040 consensus_queue.cc:237] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } }
I20260923 23:13:46.357354 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 2970
I20260923 23:13:46.357450 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:46.357474 3042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 609a950c305e494d893c585e8fe379cc. Latest consensus state: current_term: 1 leader_uuid: "609a950c305e494d893c585e8fe379cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } } }
I20260923 23:13:46.357534 3042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:46.357564 3041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "609a950c305e494d893c585e8fe379cc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "609a950c305e494d893c585e8fe379cc" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46173 } } }
I20260923 23:13:46.357638 3041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:46.357649 16874 external_mini_cluster.cc:1428] Setting key 8066fb2d851f92b27671d6e4a59736c0
I20260923 23:13:46.357752 3048 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:46.358232 3048 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:46.359788 3048 catalog_manager.cc:1347] Generated new cluster ID: b10db8f0abe044d283f0c9e2ebae3e05
I20260923 23:13:46.359841 3048 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:46.364646 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.358142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37626 (local address 127.16.122.190:46173)
0923 23:13:46.359107 (+ 965us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.359117 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.359154 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:13:46.359388 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.359395 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.359475 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.359617 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.359899 (+ 282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.361236 (+ 1337us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.362298 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.362958 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.363116 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.363780 (+ 664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.363784 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.363785 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:46.364015 (+ 230us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.364071 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:46.364095 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.364163 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.364235 (+ 72us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.364421 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":870,"thread_start_us":81,"threads_started":1}
I20260923 23:13:46.400820 3048 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:46.401332 3048 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:46.410315 3048 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: Generated new TSK 0
I20260923 23:13:46.410514 3048 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:46.413362 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--builtin_ntp_servers=127.16.122.148:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:46.491846 3061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:46.492053 3061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:46.492075 3061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:46.493162 3061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:46.493193 3061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:46.493216 3061 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:46.494490 3061 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3061
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:46.494681 3061 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:46.494927 3061 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:46.499593 3066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.499619 3061 server_base.cc:1034] running on GCE node
W20260923 23:13:46.499647 3067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.499593 3069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.499958 3061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:46.500190 3061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:46.501323 3061 hybrid_clock.cc:648] HybridClock initialized: now 1790205226501306 us; error 34 us; skew 500 ppm
I20260923 23:13:46.502530 3061 webserver.cc:460] Webserver started at http://127.16.122.129:45253/ using document root <none> and password file <none>
I20260923 23:13:46.502748 3061 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:46.502794 3061 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:46.502898 3061 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:46.503949 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "c5666a79c2bf48ed9c581f38e0c53a0c"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "ebf558d2744ac28f9f4f1ba78a3d24a6"
server_key_iv: "f610cbed3eda53e77b57a0183cd95bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.504266 3061 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "c5666a79c2bf48ed9c581f38e0c53a0c"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "ebf558d2744ac28f9f4f1ba78a3d24a6"
server_key_iv: "f610cbed3eda53e77b57a0183cd95bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.505438 3061 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:46.506058 3075 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.506207 3061 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:46.506289 3061 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "c5666a79c2bf48ed9c581f38e0c53a0c"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "ebf558d2744ac28f9f4f1ba78a3d24a6"
server_key_iv: "f610cbed3eda53e77b57a0183cd95bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.506348 3061 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:46.537222 3061 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:46.537557 3061 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:46.537695 3061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:46.537914 3061 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:46.538275 3061 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:46.538311 3061 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.538332 3061 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:46.538349 3061 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.543826 3061 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:37183
I20260923 23:13:46.543875 3188 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:37183 every 8 connection(s)
I20260923 23:13:46.544195 3061 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:46.548254 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3061
I20260923 23:13:46.548344 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:46.548522 16874 external_mini_cluster.cc:1428] Setting key c1df72f85e60e8a5b565318da0170e8c
I20260923 23:13:46.549227 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.544860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:50909 (local address 127.16.122.190:46173)
0923 23:13:46.544913 (+ 53us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.544914 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.545120 (+ 206us) server_negotiation.cc:407] Connection header received
0923 23:13:46.545623 (+ 503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.545625 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.545638 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.545694 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.546332 (+ 638us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.546836 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.547857 (+ 1021us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.548042 (+ 185us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.548104 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.548952 (+ 848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.548954 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.548954 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:46.548959 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.548976 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:46.548980 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.549039 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.549112 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.549147 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27}
I20260923 23:13:46.549427 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.544847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:46173 (local address 127.16.122.129:50909)
0923 23:13:46.545012 (+ 165us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.545051 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.545544 (+ 493us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.545754 (+ 210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.545758 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.545766 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:46.546268 (+ 502us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.546272 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.546919 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.546920 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.547795 (+ 875us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.547798 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.548105 (+ 307us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.548105 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.548210 (+ 105us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.548836 (+ 626us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:46.548837 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:46.548866 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:46.548890 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:46.549040 (+ 150us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:46.549043 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:46.549046 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.549112 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.549208 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":107,"thread_start_us":82,"threads_started":1}
I20260923 23:13:46.549556 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--builtin_ntp_servers=127.16.122.148:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:46.549911 3189 heartbeater.cc:347] Connected to a master server at 127.16.122.190:46173
I20260923 23:13:46.550006 3189 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:46.550186 3189 heartbeater.cc:511] Master 127.16.122.190:46173 requested a full tablet report, sending...
I20260923 23:13:46.550683 3001 ts_manager.cc:194] Registered new tserver with Master: c5666a79c2bf48ed9c581f38e0c53a0c (127.16.122.129:37183)
I20260923 23:13:46.551357 3001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.129:50909
W20260923 23:13:46.629266 3192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:46.629503 3192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:46.629527 3192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:46.630700 3192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:46.630738 3192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:46.630762 3192 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:46.632088 3192 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3192
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:46.632298 3192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:46.632551 3192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:46.637369 3197 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.637382 3198 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.637506 3192 server_base.cc:1034] running on GCE node
W20260923 23:13:46.637538 3200 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.637766 3192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:46.638000 3192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:46.639128 3192 hybrid_clock.cc:648] HybridClock initialized: now 1790205226639110 us; error 37 us; skew 500 ppm
I20260923 23:13:46.640385 3192 webserver.cc:460] Webserver started at http://127.16.122.130:37261/ using document root <none> and password file <none>
I20260923 23:13:46.640620 3192 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:46.640682 3192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:46.640816 3192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:46.641902 3192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "f17ac4964d7fd5e9cd1d0c8d76dbc560"
server_key_iv: "cd5497c057cabc22b672683bc26cce09"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.642237 3192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "f17ac4964d7fd5e9cd1d0c8d76dbc560"
server_key_iv: "cd5497c057cabc22b672683bc26cce09"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.643548 3192 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:46.644217 3206 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.644376 3192 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:46.644459 3192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "f17ac4964d7fd5e9cd1d0c8d76dbc560"
server_key_iv: "cd5497c057cabc22b672683bc26cce09"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.644518 3192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:46.656920 3192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:46.657235 3192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:46.657367 3192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:46.657590 3192 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:46.657912 3192 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:46.657943 3192 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.657972 3192 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:46.657996 3192 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.663287 3192 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:45785
I20260923 23:13:46.663388 3319 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:45785 every 8 connection(s)
I20260923 23:13:46.663652 3192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:46.664324 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3192
I20260923 23:13:46.664422 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:46.664678 16874 external_mini_cluster.cc:1428] Setting key db50eebc6755ffc3e73726a75cf1ef4a
I20260923 23:13:46.665643 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--builtin_ntp_servers=127.16.122.148:40263
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:46.669581 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.664337 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:57875 (local address 127.16.122.190:46173)
0923 23:13:46.664388 (+ 51us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.664390 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.664664 (+ 274us) server_negotiation.cc:407] Connection header received
0923 23:13:46.665280 (+ 616us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.665282 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.665293 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.665337 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.666075 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.666754 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.668169 (+ 1415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.668359 (+ 190us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.668415 (+ 56us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.669328 (+ 913us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.669330 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.669330 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:46.669334 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.669348 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:46.669352 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.669392 (+ 40us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.669494 (+ 102us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.669520 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:13:46.669855 3322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.664342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:46173 (local address 127.16.122.130:57875)
0923 23:13:46.664532 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.664579 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.665186 (+ 607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.665411 (+ 225us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.665415 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.665426 (+ 11us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:46.665993 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.665998 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.666860 (+ 862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.666861 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.668082 (+ 1221us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.668085 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.668433 (+ 348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.668434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.668620 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.669178 (+ 558us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:46.669179 (+ 1us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:46.669220 (+ 41us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:46.669250 (+ 30us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:46.669412 (+ 162us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:46.669417 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:46.669420 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.669500 (+ 80us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.669622 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":83,"threads_started":1}
I20260923 23:13:46.670472 3320 heartbeater.cc:347] Connected to a master server at 127.16.122.190:46173
I20260923 23:13:46.670575 3320 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:46.670783 3320 heartbeater.cc:511] Master 127.16.122.190:46173 requested a full tablet report, sending...
I20260923 23:13:46.671186 3001 ts_manager.cc:194] Registered new tserver with Master: fcf8444dc6f94875bf1a43cf3a0c0c34 (127.16.122.130:45785)
I20260923 23:13:46.671545 3001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.130:57875
W20260923 23:13:46.748301 3323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:46.748508 3323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:46.748529 3323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:46.749720 3323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:46.749763 3323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:46.749790 3323 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:46.751241 3323 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40263
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:46173
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3323
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:46.751473 3323 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:46.751755 3323 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:46.756531 3328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:46.756567 3329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.756701 3323 server_base.cc:1034] running on GCE node
W20260923 23:13:46.756872 3331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:46.757167 3323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:46.757472 3323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:46.758618 3323 hybrid_clock.cc:648] HybridClock initialized: now 1790205226758595 us; error 37 us; skew 500 ppm
I20260923 23:13:46.759828 3323 webserver.cc:460] Webserver started at http://127.16.122.131:43593/ using document root <none> and password file <none>
I20260923 23:13:46.760052 3323 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:46.760102 3323 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:46.760241 3323 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:46.761294 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "d84b3bba63df9dfabf3e0702d3242abf"
server_key_iv: "a4785cb054c1eca659265847dd6c8d3c"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.761617 3323 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "d84b3bba63df9dfabf3e0702d3242abf"
server_key_iv: "a4785cb054c1eca659265847dd6c8d3c"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.762807 3323 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:46.763518 3337 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.763667 3323 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.763754 3323 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7"
format_stamp: "Formatted at 2026-09-23 23:13:46 on dist-test-slave-95hh"
server_key: "d84b3bba63df9dfabf3e0702d3242abf"
server_key_iv: "a4785cb054c1eca659265847dd6c8d3c"
server_key_version: "encryptionkey@0"
I20260923 23:13:46.763814 3323 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:46.791018 3323 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:46.791380 3323 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:46.791504 3323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:46.791703 3323 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:46.792012 3323 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:46.792044 3323 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.792073 3323 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:46.792096 3323 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:46.797341 3323 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:41243
I20260923 23:13:46.797395 3450 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:41243 every 8 connection(s)
I20260923 23:13:46.797709 3323 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:13:46.800745 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3323
I20260923 23:13:46.800832 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:46.801002 16874 external_mini_cluster.cc:1428] Setting key f261119049f5b7d095142d28f90e0095
I20260923 23:13:46.802695 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.798380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:44929 (local address 127.16.122.190:46173)
0923 23:13:46.798444 (+ 64us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.798446 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.798653 (+ 207us) server_negotiation.cc:407] Connection header received
0923 23:13:46.799145 (+ 492us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.799146 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.799157 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.799207 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.799860 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.800344 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.801426 (+ 1082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.801615 (+ 189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.801676 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.802425 (+ 749us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.802426 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.802426 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:46.802431 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.802447 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:46.802451 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.802496 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.802575 (+ 79us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.802607 (+ 32us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:13:46.802883 3453 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.798364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:46173 (local address 127.16.122.131:44929)
0923 23:13:46.798536 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.798570 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.799066 (+ 496us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.799279 (+ 213us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.799283 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.799291 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:46.799795 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.799799 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.800407 (+ 608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.800407 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.801321 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.801324 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.801707 (+ 383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.801708 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.801836 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.802292 (+ 456us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:46.802292 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:46.802320 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:46.802343 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:46.802503 (+ 160us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:46.802508 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:46.802511 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.802571 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.802672 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":77,"threads_started":1}
I20260923 23:13:46.803321 3451 heartbeater.cc:347] Connected to a master server at 127.16.122.190:46173
I20260923 23:13:46.803401 3451 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:46.803570 3451 heartbeater.cc:511] Master 127.16.122.190:46173 requested a full tablet report, sending...
I20260923 23:13:46.803903 3001 ts_manager.cc:194] Registered new tserver with Master: fb227b94bfcd4dc6b4da1e7065eeddb7 (127.16.122.131:41243)
I20260923 23:13:46.804263 3001 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.131:44929
I20260923 23:13:46.812536 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:46.816743 3057 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.813891 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37638 (local address 127.16.122.190:46173)
0923 23:13:46.813983 (+ 92us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.813984 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.814038 (+ 54us) server_negotiation.cc:407] Connection header received
0923 23:13:46.814082 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.814082 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.814093 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.814157 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:46.814285 (+ 128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.814776 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.815579 (+ 803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.815775 (+ 196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.815841 (+ 66us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.816477 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:46.816479 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:46.816479 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:46.816483 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:46.816503 (+ 20us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:46.816509 (+ 6us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:46.816599 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.816672 (+ 73us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.816683 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260923 23:13:46.818320 3001 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:37638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:46.819878 3001 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:46.825728 3460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.821969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:45785 (local address 127.0.0.1:38442)
0923 23:13:46.822136 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.822174 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.822240 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.822539 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.822541 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.822544 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.822617 (+ 73us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.822618 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.823616 (+ 998us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.823617 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.824529 (+ 912us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.824531 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.825421 (+ 890us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.825422 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.825501 (+ 79us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.825517 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.825593 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.825644 (+ 51us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":111,"threads_started":1}
I20260923 23:13:46.826309 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.822066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38442 (local address 127.16.122.130:45785)
0923 23:13:46.822390 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.822394 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.822403 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:46.822439 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.822441 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.822455 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.822523 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.822700 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.823540 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.824624 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.825353 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.825417 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.825504 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.826169 (+ 665us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.826217 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":278,"thread_start_us":81,"threads_started":1}
I20260923 23:13:46.826372 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.822430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46362 (local address 127.16.122.129:37183)
0923 23:13:46.822660 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.822664 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.822674 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:13:46.822747 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.822749 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.822763 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.822827 (+ 64us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.823010 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.824230 (+ 1220us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.825318 (+ 1088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.826043 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.826111 (+ 68us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.826192 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.826256 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.826294 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193,"thread_start_us":85,"threads_started":1}
I20260923 23:13:46.826376 3459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.821964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:41243 (local address 127.0.0.1:51574)
0923 23:13:46.822134 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.822173 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.822240 (+ 67us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.822842 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.822843 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.822845 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.822903 (+ 58us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.822905 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.824260 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.824261 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.825338 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.825341 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.826163 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.826164 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.826214 (+ 50us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.826216 (+ 2us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.826243 (+ 27us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.826286 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134,"thread_start_us":57,"threads_started":1}
I20260923 23:13:46.826378 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.822198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:37183 (local address 127.0.0.1:46362)
0923 23:13:46.822609 (+ 411us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.822620 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.822664 (+ 44us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.822890 (+ 226us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.822891 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.822893 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.822930 (+ 37us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.822932 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.824318 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.824319 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.825233 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.825235 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.826131 (+ 896us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.826132 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.826191 (+ 59us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.826195 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.826243 (+ 48us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.826286 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":392,"mutex_wait_us":3,"thread_start_us":80,"threads_started":1}
I20260923 23:13:46.826783 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.822250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51574 (local address 127.16.122.131:41243)
0923 23:13:46.822413 (+ 163us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.822418 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.822429 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:13:46.822475 (+ 46us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.822477 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.822494 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.822986 (+ 492us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.823078 (+ 92us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.824165 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.825543 (+ 1378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.826064 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.826545 (+ 481us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.826644 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.826669 (+ 25us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.826711 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":121,"thread_start_us":80,"threads_started":1}
I20260923 23:13:46.827132 3123 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71b58c66ec4cb7be5054530efd691b (DEFAULT_TABLE table=test-table [id=39ec50afe6f7436192087b0ff527727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.827486 3123 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71b58c66ec4cb7be5054530efd691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.827476 3254 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71b58c66ec4cb7be5054530efd691b (DEFAULT_TABLE table=test-table [id=39ec50afe6f7436192087b0ff527727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.827487 3385 tablet_service.cc:1467] Processing CreateTablet for tablet 4c71b58c66ec4cb7be5054530efd691b (DEFAULT_TABLE table=test-table [id=39ec50afe6f7436192087b0ff527727a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:46.827765 3254 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71b58c66ec4cb7be5054530efd691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.827777 3385 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4c71b58c66ec4cb7be5054530efd691b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:46.830464 3465 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Bootstrap starting.
I20260923 23:13:46.830848 3466 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Bootstrap starting.
I20260923 23:13:46.830955 3467 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Bootstrap starting.
I20260923 23:13:46.831357 3465 tablet_bootstrap.cc:654] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.831526 3467 tablet_bootstrap.cc:654] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.831584 3466 tablet_bootstrap.cc:654] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:46.831681 3465 log.cc:824] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.831815 3467 log.cc:824] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.831897 3466 log.cc:824] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:46.832508 3467 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: No bootstrap required, opened a new log
I20260923 23:13:46.832525 3465 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: No bootstrap required, opened a new log
I20260923 23:13:46.832569 3467 ts_tablet_manager.cc:1397] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:46.832582 3465 ts_tablet_manager.cc:1397] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:46.832532 3466 tablet_bootstrap.cc:492] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: No bootstrap required, opened a new log
I20260923 23:13:46.832749 3466 ts_tablet_manager.cc:1397] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:46.834535 3467 raft_consensus.cc:348] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.834676 3467 raft_consensus.cc:374] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.834700 3467 raft_consensus.cc:729] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb227b94bfcd4dc6b4da1e7065eeddb7, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.834800 3467 consensus_queue.cc:260] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.835107 3467 ts_tablet_manager.cc:1428] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:46.835126 3451 heartbeater.cc:503] Master 127.16.122.190:46173 was elected leader, sending a full tablet report...
I20260923 23:13:46.834996 3465 raft_consensus.cc:348] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.835180 3465 raft_consensus.cc:374] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.835227 3465 raft_consensus.cc:729] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fcf8444dc6f94875bf1a43cf3a0c0c34, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.835345 3465 consensus_queue.cc:260] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.835582 3465 ts_tablet_manager.cc:1428] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:46.835732 3320 heartbeater.cc:503] Master 127.16.122.190:46173 was elected leader, sending a full tablet report...
I20260923 23:13:46.835928 3466 raft_consensus.cc:348] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.836084 3466 raft_consensus.cc:374] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:46.836122 3466 raft_consensus.cc:729] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c5666a79c2bf48ed9c581f38e0c53a0c, State: Initialized, Role: FOLLOWER
I20260923 23:13:46.836230 3466 consensus_queue.cc:260] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.836476 3466 ts_tablet_manager.cc:1428] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260923 23:13:46.836524 3189 heartbeater.cc:503] Master 127.16.122.190:46173 was elected leader, sending a full tablet report...
I20260923 23:13:46.903578 3473 raft_consensus.cc:482] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:46.903700 3473 raft_consensus.cc:504] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.904022 3473 leader_election.cc:288] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fcf8444dc6f94875bf1a43cf3a0c0c34 (127.16.122.130:45785), fb227b94bfcd4dc6b4da1e7065eeddb7 (127.16.122.131:41243)
I20260923 23:13:46.906697 3461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.904305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:34501 (local address 127.16.122.130:45785)
0923 23:13:46.904362 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.904364 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.904374 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:13:46.904404 (+ 30us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.904405 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.904417 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.904472 (+ 55us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.904622 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.905086 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906031 (+ 945us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.906391 (+ 360us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906454 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.906463 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.906583 (+ 120us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.906631 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:46.906697 3191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.904193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:45785 (local address 127.16.122.129:34501)
0923 23:13:46.904244 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.904257 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.904323 (+ 66us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.904489 (+ 166us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.904490 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.904492 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.904555 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.904557 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.905173 (+ 616us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.905174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.905944 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.905947 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.906466 (+ 519us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906467 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.906523 (+ 56us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.906526 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.906575 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.906618 (+ 43us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 23:13:46.906818 3474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.904194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:41243 (local address 127.16.122.129:47331)
0923 23:13:46.904419 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.904428 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:46.904468 (+ 40us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:46.904590 (+ 122us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:46.904591 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:46.904593 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:46.904660 (+ 67us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.904661 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.905274 (+ 613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.905275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.906037 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:46.906039 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.906561 (+ 522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906562 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:46.906633 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.906636 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:46.906679 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:13:46.906715 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":210,"mutex_wait_us":59,"thread_start_us":96,"threads_started":1}
I20260923 23:13:46.906803 3463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.904298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:47331 (local address 127.16.122.131:41243)
0923 23:13:46.904371 (+ 73us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.904373 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.904476 (+ 103us) server_negotiation.cc:407] Connection header received
0923 23:13:46.904527 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.904528 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.904537 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.904582 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:46.904717 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.905200 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906122 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.906491 (+ 369us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.906551 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.906558 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.906693 (+ 135us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.906735 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260923 23:13:46.907083 3274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71b58c66ec4cb7be5054530efd691b" candidate_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" is_pre_election: true
I20260923 23:13:46.907171 3405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71b58c66ec4cb7be5054530efd691b" candidate_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" is_pre_election: true
I20260923 23:13:46.907245 3274 raft_consensus.cc:2454] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5666a79c2bf48ed9c581f38e0c53a0c in term 0.
I20260923 23:13:46.907330 3405 raft_consensus.cc:2454] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c5666a79c2bf48ed9c581f38e0c53a0c in term 0.
I20260923 23:13:46.907481 3076 leader_election.cc:302] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5666a79c2bf48ed9c581f38e0c53a0c, fcf8444dc6f94875bf1a43cf3a0c0c34; no voters:
I20260923 23:13:46.907621 3473 raft_consensus.cc:2781] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:46.907670 3473 raft_consensus.cc:482] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:46.907686 3473 raft_consensus.cc:3037] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.908357 3473 raft_consensus.cc:504] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.908494 3473 leader_election.cc:288] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [CANDIDATE]: Term 1 election: Requested vote from peers fcf8444dc6f94875bf1a43cf3a0c0c34 (127.16.122.130:45785), fb227b94bfcd4dc6b4da1e7065eeddb7 (127.16.122.131:41243)
I20260923 23:13:46.908643 3405 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71b58c66ec4cb7be5054530efd691b" candidate_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7"
I20260923 23:13:46.908667 3274 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4c71b58c66ec4cb7be5054530efd691b" candidate_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34"
I20260923 23:13:46.908715 3405 raft_consensus.cc:3037] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.908736 3274 raft_consensus.cc:3037] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:46.909390 3274 raft_consensus.cc:2454] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5666a79c2bf48ed9c581f38e0c53a0c in term 1.
I20260923 23:13:46.909391 3405 raft_consensus.cc:2454] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c5666a79c2bf48ed9c581f38e0c53a0c in term 1.
I20260923 23:13:46.909562 3076 leader_election.cc:302] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: c5666a79c2bf48ed9c581f38e0c53a0c, fcf8444dc6f94875bf1a43cf3a0c0c34; no voters:
I20260923 23:13:46.909657 3473 raft_consensus.cc:2781] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:46.909793 3473 raft_consensus.cc:686] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 1 LEADER]: Becoming Leader. State: Replica: c5666a79c2bf48ed9c581f38e0c53a0c, State: Running, Role: LEADER
I20260923 23:13:46.909870 3473 consensus_queue.cc:237] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } }
I20260923 23:13:46.910717 3001 catalog_manager.cc:5495] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c reported cstate change: term changed from 0 to 1, leader changed from <none> to c5666a79c2bf48ed9c581f38e0c53a0c (127.16.122.129). New cstate: current_term: 1 leader_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "c5666a79c2bf48ed9c581f38e0c53a0c" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 37183 } health_report { overall_health: HEALTHY } } }
W20260923 23:13:46.914144 3321 tablet.cc:2369] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:13:46.917497 3464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.915352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46374 (local address 127.16.122.129:37183)
0923 23:13:46.915404 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:46.915406 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:46.915414 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:46.915467 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:46.915467 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:46.915489 (+ 22us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:46.915554 (+ 65us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:46.915668 (+ 114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.916115 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.916861 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:46.917068 (+ 207us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:46.917122 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:46.917217 (+ 95us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:46.917301 (+ 84us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:46.917364 (+ 63us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:46.917384 (+ 20us) server_negotiation.cc:299] Negotiation successful
0923 23:13:46.917392 (+ 8us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:13:46.919183 3274 raft_consensus.cc:1261] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 1 FOLLOWER]: Refusing update from remote peer c5666a79c2bf48ed9c581f38e0c53a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:46.919184 3405 raft_consensus.cc:1261] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 1 FOLLOWER]: Refusing update from remote peer c5666a79c2bf48ed9c581f38e0c53a0c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:46.919504 3475 consensus_queue.cc:1035] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fcf8444dc6f94875bf1a43cf3a0c0c34" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 45785 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:46.919613 3473 consensus_queue.cc:1035] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb227b94bfcd4dc6b4da1e7065eeddb7" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 41243 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:46.921684 3478 mvcc.cc:205] Tried to move back new op lower bound from 7332680609458589696 to 7332680609423642624. Current Snapshot: MvccSnapshot[applied={T|T < 7332680609458589696}]
I20260923 23:13:46.922153 3479 mvcc.cc:205] Tried to move back new op lower bound from 7332680609458589696 to 7332680609423642624. Current Snapshot: MvccSnapshot[applied={T|T < 7332680609458589696}]
I20260923 23:13:46.922466 3480 mvcc.cc:205] Tried to move back new op lower bound from 7332680609458589696 to 7332680609423642624. Current Snapshot: MvccSnapshot[applied={T|T < 7332680609458589696}]
I20260923 23:13:46.924261 3001 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:46.924353 3001 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:37638:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:46.925083 3001 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 609a950c305e494d893c585e8fe379cc: Sending DeleteTablet for 3 replicas of tablet 4c71b58c66ec4cb7be5054530efd691b
I20260923 23:13:46.925446 3123 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71b58c66ec4cb7be5054530efd691b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='slave'} at 127.0.0.1:46362
I20260923 23:13:46.925486 3385 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71b58c66ec4cb7be5054530efd691b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='slave'} at 127.0.0.1:51574
I20260923 23:13:46.925639 3488 tablet_replica.cc:330] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: stopping tablet replica
I20260923 23:13:46.925693 3254 tablet_service.cc:1514] Processing DeleteTablet for tablet 4c71b58c66ec4cb7be5054530efd691b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:46 UTC) from {username='slave'} at 127.0.0.1:38442
I20260923 23:13:46.925727 3488 raft_consensus.cc:2229] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:46.925839 3488 raft_consensus.cc:2258] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:46.925877 3490 tablet_replica.cc:330] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: stopping tablet replica
I20260923 23:13:46.925971 3490 raft_consensus.cc:2229] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:46.926118 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3061
I20260923 23:13:46.926163 3489 tablet_replica.cc:330] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: stopping tablet replica
I20260923 23:13:46.926256 3488 ts_tablet_manager.cc:1905] T 4c71b58c66ec4cb7be5054530efd691b P c5666a79c2bf48ed9c581f38e0c53a0c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:46.926678 3489 raft_consensus.cc:2229] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:46.926786 3489 raft_consensus.cc:2258] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:46.926951 3490 raft_consensus.cc:2258] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:46.927343 3489 ts_tablet_manager.cc:1905] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:46.927374 3490 ts_tablet_manager.cc:1905] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:46.928764 3490 ts_tablet_manager.cc:1918] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:46.928812 3490 log.cc:1196] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/4c71b58c66ec4cb7be5054530efd691b
I20260923 23:13:46.928963 3490 ts_tablet_manager.cc:1939] T 4c71b58c66ec4cb7be5054530efd691b P fcf8444dc6f94875bf1a43cf3a0c0c34: Deleting consensus metadata
I20260923 23:13:46.929196 3489 ts_tablet_manager.cc:1918] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:46.929255 3489 log.cc:1196] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/4c71b58c66ec4cb7be5054530efd691b
I20260923 23:13:46.929392 2985 catalog_manager.cc:4841] TS fcf8444dc6f94875bf1a43cf3a0c0c34 (127.16.122.130:45785): tablet 4c71b58c66ec4cb7be5054530efd691b (table test-table [id=39ec50afe6f7436192087b0ff527727a]) successfully deleted
I20260923 23:13:46.929556 3489 ts_tablet_manager.cc:1939] T 4c71b58c66ec4cb7be5054530efd691b P fb227b94bfcd4dc6b4da1e7065eeddb7: Deleting consensus metadata
I20260923 23:13:46.929910 2988 catalog_manager.cc:4841] TS fb227b94bfcd4dc6b4da1e7065eeddb7 (127.16.122.131:41243): tablet 4c71b58c66ec4cb7be5054530efd691b (table test-table [id=39ec50afe6f7436192087b0ff527727a]) successfully deleted
W20260923 23:13:46.932727 2985 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:37183 (error 108)
I20260923 23:13:46.932791 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3192
I20260923 23:13:46.933645 3462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:46.933525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:37183 (local address 127.0.0.1:46384)
0923 23:13:46.933575 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:46.933615 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:37183: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:13:46.933758 2985 catalog_manager.cc:4584] TS c5666a79c2bf48ed9c581f38e0c53a0c (127.16.122.129:37183): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4c71b58c66ec4cb7be5054530efd691b: Network error: Client connection negotiation failed: client connection to 127.16.122.129:37183: connect: Connection refused (error 111)
I20260923 23:13:46.937443 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3323
I20260923 23:13:46.941488 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 2970
2026-09-23T23:13:46Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (732 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:46 dist-test-slave-95hh krb5kdc[3494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:46 dist-test-slave-95hh krb5kdc[3494](info): set up 2 sockets
Sep 23 23:13:46 dist-test-slave-95hh krb5kdc[3494](info): commencing operation
krb5kdc: starting...
W20260923 23:13:47.898597 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.939s user 0.001s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:47 dist-test-slave-95hh krb5kdc[3494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:13:48.009550 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34639
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34639
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:48.087129 3507 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.087328 3507 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.087349 3507 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.088419 3507 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:48.088450 3507 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.088462 3507 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.088474 3507 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:48.088487 3507 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:48.089857 3507 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34639
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3507
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.090047 3507 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.090219 3507 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:48.094808 3512 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.094878 3515 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.094916 3507 server_base.cc:1034] running on GCE node
W20260923 23:13:48.094825 3513 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.095321 3507 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.095598 3507 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.096730 3507 hybrid_clock.cc:648] HybridClock initialized: now 1790205228096718 us; error 27 us; skew 500 ppm
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:48.103081 3507 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:13:48.103521 3507 webserver.cc:460] Webserver started at http://127.16.122.190:33969/ using document root <none> and password file <none>
I20260923 23:13:48.103716 3507 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.103756 3507 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.103847 3507 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.104863 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "0caaba265040443c897a490b77786936"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "6074612b12602e22f66ac6aeb877f82e"
server_key_iv: "adef901f6ab4b886b6e95cbc74892fe9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.105163 3507 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "0caaba265040443c897a490b77786936"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "6074612b12602e22f66ac6aeb877f82e"
server_key_iv: "adef901f6ab4b886b6e95cbc74892fe9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.106303 3507 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:48.106931 3522 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.107095 3507 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:48.107175 3507 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "0caaba265040443c897a490b77786936"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "6074612b12602e22f66ac6aeb877f82e"
server_key_iv: "adef901f6ab4b886b6e95cbc74892fe9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.107239 3507 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.120967 3507 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.123131 3507 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.123270 3507 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.126942 3507 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34639
I20260923 23:13:48.126987 3574 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34639 every 8 connection(s)
I20260923 23:13:48.127357 3507 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:48.128163 3575 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:48.130371 3575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: Bootstrap starting.
I20260923 23:13:48.130920 3575 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:48.131210 3575 log.cc:824] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:48.131843 3575 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: No bootstrap required, opened a new log
I20260923 23:13:48.133399 3575 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } }
I20260923 23:13:48.133553 3575 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:48.133575 3575 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0caaba265040443c897a490b77786936, State: Initialized, Role: FOLLOWER
I20260923 23:13:48.133706 3575 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } }
I20260923 23:13:48.133788 3575 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:48.133827 3575 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:48.133879 3575 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:48.133924 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3507
I20260923 23:13:48.134018 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:48.134202 16874 external_mini_cluster.cc:1428] Setting key 4a5e4b01384a0408dc40ec84925dd204
I20260923 23:13:48.134524 3575 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } }
I20260923 23:13:48.134625 3575 leader_election.cc:302] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0caaba265040443c897a490b77786936; no voters:
I20260923 23:13:48.134783 3575 leader_election.cc:288] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:48.134922 3581 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:48.135021 3575 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:48.135094 3581 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [term 1 LEADER]: Becoming Leader. State: Replica: 0caaba265040443c897a490b77786936, State: Running, Role: LEADER
I20260923 23:13:48.135180 3581 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } }
I20260923 23:13:48.135727 3580 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0caaba265040443c897a490b77786936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } } }
I20260923 23:13:48.135805 3580 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:48.135951 3582 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0caaba265040443c897a490b77786936. Latest consensus state: current_term: 1 leader_uuid: "0caaba265040443c897a490b77786936" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0caaba265040443c897a490b77786936" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34639 } } }
I20260923 23:13:48.136018 3582 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936 [sys.catalog]: This master's current role is: LEADER
W20260923 23:13:48.136783 3596 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:13:48.136837 3596 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:13:48.136889 3589 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:48.137046 3589 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:48.138598 3589 catalog_manager.cc:1347] Generated new cluster ID: 7bb59c6f91ad4301a4fe0bce44f9e50d
I20260923 23:13:48.138650 3589 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:48.145362 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.134660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43900 (local address 127.16.122.190:34639)
0923 23:13:48.134838 (+ 178us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.134848 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.134884 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:13:48.135008 (+ 124us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.135017 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.135105 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.136019 (+ 914us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.136134 (+ 115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.137553 (+ 1419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.138448 (+ 895us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.139087 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.139201 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.141848 (+ 2647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.141852 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.141853 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:48.141865 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.143663 (+ 1798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.144056 (+ 393us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.144059 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.144059 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.144166 (+ 107us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.144352 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.144353 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.144353 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.144629 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.144774 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.144896 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.145036 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124,"thread_start_us":88,"threads_started":1}
I20260923 23:13:48.155575 3589 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:48.156337 3589 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:48.162387 3589 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: Generated new TSK 0
I20260923 23:13:48.162585 3589 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.122.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
Entry for principal kudu/127.16.122.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab.
I20260923 23:13:48.191007 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:48.272485 3601 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.272698 3601 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.272729 3601 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.273892 3601 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.273936 3601 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.273972 3601 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:48.275488 3601 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.129.keytab
--principal=kudu/127.16.122.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3601
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.275740 3601 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.275967 3601 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:48.280721 3607 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.280715 3609 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.280716 3606 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.281013 3601 server_base.cc:1034] running on GCE node
I20260923 23:13:48.281203 3601 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.281459 3601 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.282593 3601 hybrid_clock.cc:648] HybridClock initialized: now 1790205228282575 us; error 34 us; skew 500 ppm
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:48.284893 3601 init.cc:377] Logged in from keytab as kudu/127.16.122.129@KRBTEST.COM (short username kudu)
I20260923 23:13:48.285310 3601 webserver.cc:460] Webserver started at http://127.16.122.129:39043/ using document root <none> and password file <none>
I20260923 23:13:48.285534 3601 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.285583 3601 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.285691 3601 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.286698 3601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "46206ef3aa5e4db09619d7d242f76c57"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "5521fa333f36a564d294f904f83ad96c"
server_key_iv: "3f2e39d436e442a12e92498b890a55ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.287005 3601 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "46206ef3aa5e4db09619d7d242f76c57"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "5521fa333f36a564d294f904f83ad96c"
server_key_iv: "3f2e39d436e442a12e92498b890a55ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.288213 3601 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:48.288853 3616 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.288998 3601 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.289073 3601 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "46206ef3aa5e4db09619d7d242f76c57"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "5521fa333f36a564d294f904f83ad96c"
server_key_iv: "3f2e39d436e442a12e92498b890a55ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.289130 3601 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.305544 3601 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.307550 3601 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.307694 3601 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.307901 3601 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:48.308187 3601 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:48.308218 3601 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.308245 3601 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:48.308266 3601 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.313733 3601 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:36027
I20260923 23:13:48.313786 3729 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:36027 every 8 connection(s)
I20260923 23:13:48.314072 3601 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:48.316432 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3601
I20260923 23:13:48.316551 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:48.316752 16874 external_mini_cluster.cc:1428] Setting key 7f0bd019151c8f4ef8bed32ed210f346
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.129@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:48.323282 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.314684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:38477 (local address 127.16.122.190:34639)
0923 23:13:48.314768 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.314771 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.314945 (+ 174us) server_negotiation.cc:407] Connection header received
0923 23:13:48.315462 (+ 517us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.315463 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.315473 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.315523 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.316448 (+ 925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.317143 (+ 695us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.318190 (+ 1047us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.318463 (+ 273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.318535 (+ 72us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.320767 (+ 2232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.320769 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.320769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:48.320780 (+ 11us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.322029 (+ 1249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.322389 (+ 360us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.322390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.322391 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.322442 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.322653 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.322654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.322655 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.322759 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.322814 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.323097 (+ 283us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.323182 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:13:48.323446 3732 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.314672 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34639 (local address 127.16.122.129:38477)
0923 23:13:48.314832 (+ 160us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.314868 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.315377 (+ 509us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.315582 (+ 205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.315585 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.316137 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:48.316341 (+ 204us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.316346 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.317228 (+ 882us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.317229 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.318100 (+ 871us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.318103 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.318555 (+ 452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.318556 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.318704 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.319201 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:48.319211 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:48.320685 (+ 1474us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:48.322100 (+ 1415us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.322106 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.322109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.322323 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.322495 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.322496 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.322496 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.322600 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.322813 (+ 213us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:48.322819 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:48.322951 (+ 132us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.323067 (+ 116us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.323235 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120,"thread_start_us":85,"threads_started":1}
I20260923 23:13:48.323912 3730 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34639
I20260923 23:13:48.323997 3730 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.122.130@KRBTEST.COM; defaulting to no policy
I20260923 23:13:48.324177 3730 heartbeater.cc:511] Master 127.16.122.190:34639 requested a full tablet report, sending...
I20260923 23:13:48.324657 3539 ts_manager.cc:194] Registered new tserver with Master: 46206ef3aa5e4db09619d7d242f76c57 (127.16.122.129:36027)
I20260923 23:13:48.325305 3539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.129@KRBTEST.COM'} at 127.16.122.129:38477
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
Entry for principal kudu/127.16.122.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab.
I20260923 23:13:48.340903 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:48.420846 3735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.421059 3735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.421082 3735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.422228 3735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.422267 3735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.422291 3735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:48.423771 3735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.130.keytab
--principal=kudu/127.16.122.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3735
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.423996 3735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.424203 3735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:48.429145 3735 server_base.cc:1034] running on GCE node
W20260923 23:13:48.429126 3740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.429277 3743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.429126 3741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.429548 3735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.429786 3735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.430925 3735 hybrid_clock.cc:648] HybridClock initialized: now 1790205228430912 us; error 28 us; skew 500 ppm
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:48.433197 3735 init.cc:377] Logged in from keytab as kudu/127.16.122.130@KRBTEST.COM (short username kudu)
I20260923 23:13:48.433606 3735 webserver.cc:460] Webserver started at http://127.16.122.130:42079/ using document root <none> and password file <none>
I20260923 23:13:48.433811 3735 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.433859 3735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.433980 3735 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.435102 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "b8c2c28dd7fa49aab00122340e51cd84"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "16f1b695e2ff245951d8651eac71b39e"
server_key_iv: "db6c8852fc98a721d9fb596bde15233a"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.435424 3735 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "b8c2c28dd7fa49aab00122340e51cd84"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "16f1b695e2ff245951d8651eac71b39e"
server_key_iv: "db6c8852fc98a721d9fb596bde15233a"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.436724 3735 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:13:48.437361 3750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.437492 3735 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:48.437572 3735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "b8c2c28dd7fa49aab00122340e51cd84"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "16f1b695e2ff245951d8651eac71b39e"
server_key_iv: "db6c8852fc98a721d9fb596bde15233a"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.437635 3735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.456369 3735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.458091 3735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.458225 3735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.458437 3735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:48.458724 3735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:48.458765 3735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.458797 3735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:48.458820 3735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.463997 3735 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:40121
I20260923 23:13:48.464058 3863 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:40121 every 8 connection(s)
I20260923 23:13:48.464336 3735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:48.465353 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3735
I20260923 23:13:48.465466 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:48.465628 16874 external_mini_cluster.cc:1428] Setting key 3cdb9cbfc8d50e737bf24f34865b99b4
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.130@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.131@KRBTEST.COM; defaulting to no policy
I20260923 23:13:48.473285 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.465021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:39905 (local address 127.16.122.190:34639)
0923 23:13:48.465076 (+ 55us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.465078 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.465278 (+ 200us) server_negotiation.cc:407] Connection header received
0923 23:13:48.465839 (+ 561us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.465840 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.465852 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.465922 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.466917 (+ 995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.467462 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.468409 (+ 947us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.468610 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.468663 (+ 53us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.470741 (+ 2078us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.470743 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.470743 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:48.470749 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.472021 (+ 1272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.472452 (+ 431us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.472453 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.472453 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.472497 (+ 44us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.472723 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.472723 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.472723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.472816 (+ 93us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.472872 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.473160 (+ 288us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.473221 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:13:48.473445 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.464998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34639 (local address 127.16.122.130:39905)
0923 23:13:48.465152 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.465194 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.465731 (+ 537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.465993 (+ 262us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.465997 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.466630 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:48.466830 (+ 200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.466835 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.467539 (+ 704us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.467540 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.468329 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.468331 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.468676 (+ 345us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.468677 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.468811 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.469257 (+ 446us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:48.469264 (+ 7us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:48.470655 (+ 1391us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:48.472098 (+ 1443us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.472105 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.472107 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.472385 (+ 278us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.472547 (+ 162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.472548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.472548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.472673 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.472875 (+ 202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:48.472881 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:48.473040 (+ 159us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.473142 (+ 102us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.473263 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":113,"thread_start_us":71,"threads_started":1}
I20260923 23:13:48.473881 3864 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34639
I20260923 23:13:48.473970 3864 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:48.474135 3864 heartbeater.cc:511] Master 127.16.122.190:34639 requested a full tablet report, sending...
I20260923 23:13:48.474448 3539 ts_manager.cc:194] Registered new tserver with Master: b8c2c28dd7fa49aab00122340e51cd84 (127.16.122.130:40121)
I20260923 23:13:48.474828 3539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.130@KRBTEST.COM'} at 127.16.122.130:39905
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
Entry for principal kudu/127.16.122.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab.
I20260923 23:13:48.487820 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:33335
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:48.565894 3869 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.566097 3869 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.566118 3869 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.567277 3869 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.567314 3869 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.567340 3869 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:48.568765 3869 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:33335
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.131.keytab
--principal=kudu/127.16.122.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:34639
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.3869
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.568990 3869 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.569195 3869 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:48.573940 3874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.573928 3877 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.573941 3875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.574280 3869 server_base.cc:1034] running on GCE node
I20260923 23:13:48.574484 3869 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.574715 3869 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.575876 3869 hybrid_clock.cc:648] HybridClock initialized: now 1790205228575864 us; error 31 us; skew 500 ppm
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:48.578127 3869 init.cc:377] Logged in from keytab as kudu/127.16.122.131@KRBTEST.COM (short username kudu)
I20260923 23:13:48.578536 3869 webserver.cc:460] Webserver started at http://127.16.122.131:41163/ using document root <none> and password file <none>
I20260923 23:13:48.578745 3869 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.578794 3869 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.578915 3869 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.579989 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "d5c7e1ed5abe453d865b45f732ef6592"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "770df3fbfd2c8de194fbff96242b0e36"
server_key_iv: "3d1bd919ba7f7ffa277bb7fe233e06bb"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.580310 3869 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "d5c7e1ed5abe453d865b45f732ef6592"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "770df3fbfd2c8de194fbff96242b0e36"
server_key_iv: "3d1bd919ba7f7ffa277bb7fe233e06bb"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.581496 3869 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:48.582144 3884 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.582284 3869 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:48.582361 3869 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "d5c7e1ed5abe453d865b45f732ef6592"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "770df3fbfd2c8de194fbff96242b0e36"
server_key_iv: "3d1bd919ba7f7ffa277bb7fe233e06bb"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.582425 3869 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.593472 3869 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.595237 3869 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.595369 3869 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.595589 3869 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:48.595876 3869 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:48.595907 3869 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.595934 3869 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:48.595952 3869 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.601280 3869 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:35419
I20260923 23:13:48.601323 3997 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:35419 every 8 connection(s)
I20260923 23:13:48.601619 3869 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:13:48.601835 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 3869
I20260923 23:13:48.601949 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:48.602147 16874 external_mini_cluster.cc:1428] Setting key 5d27d9d1d706a7cbbed1d5bc0e01241c
Sep 23 23:13:48 dist-test-slave-95hh krb5kdc[3494](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205228, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.131@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:13:48.610561 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.602334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:37561 (local address 127.16.122.190:34639)
0923 23:13:48.602397 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.602399 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.602616 (+ 217us) server_negotiation.cc:407] Connection header received
0923 23:13:48.603235 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.603237 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.603248 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.603306 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.604210 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.604706 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.605678 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.605872 (+ 194us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.605926 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.608172 (+ 2246us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.608173 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.608174 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:48.608181 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.609274 (+ 1093us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.609652 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.609653 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.609653 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.609707 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.609933 (+ 226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.609934 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.609935 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.610045 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.610100 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.610368 (+ 268us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.610447 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:13:48.610705 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.602325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:34639 (local address 127.16.122.131:37561)
0923 23:13:48.602497 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.602545 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.603146 (+ 601us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.603382 (+ 236us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.603386 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.603936 (+ 550us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:48.604124 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.604128 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.604791 (+ 663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.604792 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.605599 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.605601 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.605940 (+ 339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.605941 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.606076 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.606704 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:13:48.606713 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:48.608086 (+ 1373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:48.609344 (+ 1258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.609350 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.609353 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.609589 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.609763 (+ 174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:13:48.609764 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:13:48.609764 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:13:48.609879 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:13:48.610098 (+ 219us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:48.610105 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:48.610248 (+ 143us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.610337 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.610491 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":129,"thread_start_us":81,"threads_started":1}
I20260923 23:13:48.611151 3998 heartbeater.cc:347] Connected to a master server at 127.16.122.190:34639
I20260923 23:13:48.611245 3998 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:48.611388 3998 heartbeater.cc:511] Master 127.16.122.190:34639 requested a full tablet report, sending...
I20260923 23:13:48.611716 3539 ts_manager.cc:194] Registered new tserver with Master: d5c7e1ed5abe453d865b45f732ef6592 (127.16.122.131:35419)
I20260923 23:13:48.612051 3539 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.122.131@KRBTEST.COM'} at 127.16.122.131:37561
I20260923 23:13:48.613744 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:48.620671 3579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.615020 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43906 (local address 127.16.122.190:34639)
0923 23:13:48.615072 (+ 52us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.615074 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.615198 (+ 124us) server_negotiation.cc:407] Connection header received
0923 23:13:48.615259 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.615260 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.615269 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.615320 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.615843 (+ 523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.616317 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.617010 (+ 693us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.617202 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.617253 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.618395 (+ 1142us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.618396 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.618397 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:48.618404 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.619560 (+ 1156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.619899 (+ 339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.619900 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.619900 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.619954 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:48.620128 (+ 174us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:48.620128 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:48.620129 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:48.620284 (+ 155us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.620378 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.620499 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.620577 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20260923 23:13:48.621978 3539 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:48.623564 3539 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:48.629698 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625919 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58506 (local address 127.16.122.131:35419)
0923 23:13:48.626068 (+ 149us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.626072 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.626080 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:13:48.626109 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.626110 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.626121 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.626205 (+ 84us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.626388 (+ 183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627300 (+ 912us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.628711 (+ 1411us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629301 (+ 590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.629374 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.629432 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.629570 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.629610 (+ 40us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117,"thread_start_us":86,"threads_started":1}
I20260923 23:13:48.629685 4008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:35419 (local address 127.0.0.1:58506)
0923 23:13:48.625908 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.625919 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.625977 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.626230 (+ 253us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.626233 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.626236 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.626304 (+ 68us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.626306 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627390 (+ 1084us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.627390 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.628628 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.628631 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629379 (+ 748us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.629380 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.629458 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.629472 (+ 14us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.629537 (+ 65us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.629596 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205,"mutex_wait_us":4,"thread_start_us":79,"threads_started":1}
I20260923 23:13:48.629868 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.16.122.129:36027)
0923 23:13:48.626122 (+ 375us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.626125 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.626143 (+ 18us) server_negotiation.cc:407] Connection header received
0923 23:13:48.626176 (+ 33us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.626178 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.626194 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.626277 (+ 83us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.626451 (+ 174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627548 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.628743 (+ 1195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629377 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.629446 (+ 69us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.629494 (+ 48us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.629728 (+ 234us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.629769 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":341,"thread_start_us":72,"threads_started":1}
I20260923 23:13:48.630017 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:36027 (local address 127.0.0.1:42792)
0923 23:13:48.625756 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.625799 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.625863 (+ 64us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.626358 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.626359 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.626361 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.626393 (+ 32us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.626395 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627662 (+ 1267us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.627663 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.628686 (+ 1023us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.628688 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629528 (+ 840us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.629529 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.629592 (+ 63us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.629596 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.629643 (+ 47us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.629685 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":161,"thread_start_us":55,"threads_started":1}
I20260923 23:13:48.630359 4006 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625523 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:40121 (local address 127.0.0.1:54314)
0923 23:13:48.625812 (+ 289us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.625821 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.625862 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.626720 (+ 858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.626721 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.626723 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.626768 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.626770 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627921 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.627922 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.628835 (+ 913us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.628839 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629697 (+ 858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.629698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.630214 (+ 516us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.630219 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.630251 (+ 32us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.630298 (+ 47us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":279,"thread_start_us":62,"threads_started":1}
I20260923 23:13:48.630671 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.625712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.16.122.130:40121)
0923 23:13:48.626560 (+ 848us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.626564 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.626573 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:48.626609 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.626611 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.626627 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.626713 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.626836 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.627825 (+ 989us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.628927 (+ 1102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.629618 (+ 691us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.630464 (+ 846us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.630526 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.630552 (+ 26us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.630597 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":97,"threads_started":1}
I20260923 23:13:48.630635 3664 tablet_service.cc:1467] Processing CreateTablet for tablet 481d8be940604685ab931e58dda9048c (DEFAULT_TABLE table=test-table [id=a7a6306cf5524f47afe7ba239eae7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:48.630970 3664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d8be940604685ab931e58dda9048c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:48.630367 3932 tablet_service.cc:1467] Processing CreateTablet for tablet 481d8be940604685ab931e58dda9048c (DEFAULT_TABLE table=test-table [id=a7a6306cf5524f47afe7ba239eae7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:48.631830 3932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d8be940604685ab931e58dda9048c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:48.634528 4012 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: Bootstrap starting.
I20260923 23:13:48.635176 4012 tablet_bootstrap.cc:654] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:48.635758 4012 log.cc:824] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:48.636425 4012 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: No bootstrap required, opened a new log
I20260923 23:13:48.636482 4012 ts_tablet_manager.cc:1397] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:13:48.638240 4012 raft_consensus.cc:348] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.638399 4012 raft_consensus.cc:374] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:48.638427 4012 raft_consensus.cc:729] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46206ef3aa5e4db09619d7d242f76c57, State: Initialized, Role: FOLLOWER
I20260923 23:13:48.638537 4012 consensus_queue.cc:260] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.638799 4012 ts_tablet_manager.cc:1428] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:13:48.639665 3730 heartbeater.cc:503] Master 127.16.122.190:34639 was elected leader, sending a full tablet report...
I20260923 23:13:48.640482 4014 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Bootstrap starting.
I20260923 23:13:48.640645 3798 tablet_service.cc:1467] Processing CreateTablet for tablet 481d8be940604685ab931e58dda9048c (DEFAULT_TABLE table=test-table [id=a7a6306cf5524f47afe7ba239eae7076]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:48.640967 3798 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 481d8be940604685ab931e58dda9048c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:48.641182 4014 tablet_bootstrap.cc:654] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:48.641782 4014 log.cc:824] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:48.642494 4014 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: No bootstrap required, opened a new log
I20260923 23:13:48.642552 4014 ts_tablet_manager.cc:1397] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Time spent bootstrapping tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:48.643635 4017 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Bootstrap starting.
I20260923 23:13:48.644124 4017 tablet_bootstrap.cc:654] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:48.644034 4014 raft_consensus.cc:348] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.644228 4014 raft_consensus.cc:374] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:48.644273 4014 raft_consensus.cc:729] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5c7e1ed5abe453d865b45f732ef6592, State: Initialized, Role: FOLLOWER
I20260923 23:13:48.644387 4014 consensus_queue.cc:260] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.644618 4017 log.cc:824] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:48.644650 4014 ts_tablet_manager.cc:1428] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:48.644677 3998 heartbeater.cc:503] Master 127.16.122.190:34639 was elected leader, sending a full tablet report...
I20260923 23:13:48.645309 4017 tablet_bootstrap.cc:492] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: No bootstrap required, opened a new log
I20260923 23:13:48.645375 4017 ts_tablet_manager.cc:1397] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:48.646852 4017 raft_consensus.cc:348] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.646976 4017 raft_consensus.cc:374] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:48.646996 4017 raft_consensus.cc:729] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8c2c28dd7fa49aab00122340e51cd84, State: Initialized, Role: FOLLOWER
I20260923 23:13:48.647109 4017 consensus_queue.cc:260] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.647342 4017 ts_tablet_manager.cc:1428] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260923 23:13:48.647352 3864 heartbeater.cc:503] Master 127.16.122.190:34639 was elected leader, sending a full tablet report...
I20260923 23:13:48.670437 4020 raft_consensus.cc:482] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:48.670562 4020 raft_consensus.cc:504] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.670926 4020 leader_election.cc:288] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46206ef3aa5e4db09619d7d242f76c57 (127.16.122.129:36027), d5c7e1ed5abe453d865b45f732ef6592 (127.16.122.131:35419)
I20260923 23:13:48.671676 4018 raft_consensus.cc:482] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:48.671782 4018 raft_consensus.cc:504] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.672104 4018 leader_election.cc:288] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 46206ef3aa5e4db09619d7d242f76c57 (127.16.122.129:36027), b8c2c28dd7fa49aab00122340e51cd84 (127.16.122.130:40121)
I20260923 23:13:48.674474 3866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.671129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:36027 (local address 127.16.122.130:37613)
0923 23:13:48.671195 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.671208 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.671264 (+ 56us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.671412 (+ 148us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.671413 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.671414 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.671486 (+ 72us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.671488 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.672314 (+ 826us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.672315 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.673496 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.673498 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.674233 (+ 735us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.674233 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.674316 (+ 83us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.674320 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.674363 (+ 43us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.674399 (+ 36us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:13:48.674595 4011 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.671374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:34895 (local address 127.16.122.131:35419)
0923 23:13:48.671409 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.671411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.671502 (+ 91us) server_negotiation.cc:407] Connection header received
0923 23:13:48.671567 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.671568 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.671575 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.671620 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.671756 (+ 136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.672380 (+ 624us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.673701 (+ 1321us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.674259 (+ 558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.674321 (+ 62us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.674327 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.674498 (+ 171us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.674529 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":13}
I20260923 23:13:48.674714 4010 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.671192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:37613 (local address 127.16.122.129:36027)
0923 23:13:48.671255 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.671258 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.671269 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:13:48.671343 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.671344 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.671354 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.671399 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.671557 (+ 158us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.672226 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.673583 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.674127 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.674193 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.674202 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.674619 (+ 417us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.674654 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:13:48.674804 4021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.671238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:35419 (local address 127.16.122.130:34895)
0923 23:13:48.671439 (+ 201us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.671448 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.671506 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.671622 (+ 116us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.671623 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.671624 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.671704 (+ 80us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.671706 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.672462 (+ 756us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.672463 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.673618 (+ 1155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.673620 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.674359 (+ 739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.674360 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.674431 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.674435 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.674691 (+ 256us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.674730 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":186,"mutex_wait_us":3,"thread_start_us":103,"threads_started":1}
I20260923 23:13:48.674978 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "b8c2c28dd7fa49aab00122340e51cd84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46206ef3aa5e4db09619d7d242f76c57" is_pre_election: true
I20260923 23:13:48.675083 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "b8c2c28dd7fa49aab00122340e51cd84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c7e1ed5abe453d865b45f732ef6592" is_pre_election: true
I20260923 23:13:48.675143 3684 raft_consensus.cc:2454] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c2c28dd7fa49aab00122340e51cd84 in term 0.
I20260923 23:13:48.675199 3952 raft_consensus.cc:2454] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b8c2c28dd7fa49aab00122340e51cd84 in term 0.
I20260923 23:13:48.675348 3751 leader_election.cc:302] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b8c2c28dd7fa49aab00122340e51cd84, d5c7e1ed5abe453d865b45f732ef6592; no voters:
I20260923 23:13:48.675483 4020 raft_consensus.cc:2781] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:48.675565 4020 raft_consensus.cc:482] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:48.675590 4020 raft_consensus.cc:3037] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:48.675706 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.672471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:49623 (local address 127.16.122.130:40121)
0923 23:13:48.672505 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.672506 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.672602 (+ 96us) server_negotiation.cc:407] Connection header received
0923 23:13:48.672684 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.672685 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.672693 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.672719 (+ 26us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.672892 (+ 173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.673531 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.674896 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.675379 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.675419 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.675425 (+ 6us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.675620 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.675653 (+ 33us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:13:48.675886 4022 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.672296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:40121 (local address 127.16.122.131:49623)
0923 23:13:48.672550 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.672559 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.672622 (+ 63us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.672766 (+ 144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.672767 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.672768 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.672831 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.672832 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.673609 (+ 777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.673610 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.674807 (+ 1197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.674810 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.675478 (+ 668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.675478 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.675544 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.675547 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.675778 (+ 231us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.675816 (+ 38us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":240,"mutex_wait_us":51,"thread_start_us":88,"threads_started":1}
I20260923 23:13:48.676185 3818 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "d5c7e1ed5abe453d865b45f732ef6592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b8c2c28dd7fa49aab00122340e51cd84" is_pre_election: true
I20260923 23:13:48.676405 4000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.672220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:36027 (local address 127.16.122.131:38453)
0923 23:13:48.672296 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.672305 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:48.672334 (+ 29us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:48.672732 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:48.672733 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:48.672735 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:48.672785 (+ 50us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.672786 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.673697 (+ 911us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.673698 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.674808 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:48.674810 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.676187 (+ 1377us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.676188 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:48.676257 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.676261 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:48.676310 (+ 49us) client_negotiation.cc:240] Negotiation successful
0923 23:13:48.676349 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61,"mutex_wait_us":32}
I20260923 23:13:48.676506 4020 raft_consensus.cc:504] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.676641 4023 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.672396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:38453 (local address 127.16.122.129:36027)
0923 23:13:48.672594 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.672596 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.672603 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:48.672622 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.672623 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.672631 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.672841 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:48.672872 (+ 31us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.673584 (+ 712us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.675540 (+ 1956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.676102 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.676175 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.676184 (+ 9us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.676416 (+ 232us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.676569 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":180,"thread_start_us":76,"threads_started":1}
I20260923 23:13:48.676636 4020 leader_election.cc:288] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [CANDIDATE]: Term 1 election: Requested vote from peers 46206ef3aa5e4db09619d7d242f76c57 (127.16.122.129:36027), d5c7e1ed5abe453d865b45f732ef6592 (127.16.122.131:35419)
I20260923 23:13:48.676641 3818 raft_consensus.cc:2379] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d5c7e1ed5abe453d865b45f732ef6592 in current term 1: Already voted for candidate b8c2c28dd7fa49aab00122340e51cd84 in this term.
I20260923 23:13:48.676776 3684 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "b8c2c28dd7fa49aab00122340e51cd84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46206ef3aa5e4db09619d7d242f76c57"
I20260923 23:13:48.676807 3952 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "b8c2c28dd7fa49aab00122340e51cd84" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d5c7e1ed5abe453d865b45f732ef6592"
I20260923 23:13:48.676807 3683 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "481d8be940604685ab931e58dda9048c" candidate_uuid: "d5c7e1ed5abe453d865b45f732ef6592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "46206ef3aa5e4db09619d7d242f76c57" is_pre_election: true
I20260923 23:13:48.676862 3952 raft_consensus.cc:3037] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:48.676843 3684 raft_consensus.cc:3037] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:48.677078 3885 leader_election.cc:302] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d5c7e1ed5abe453d865b45f732ef6592; no voters: 46206ef3aa5e4db09619d7d242f76c57, b8c2c28dd7fa49aab00122340e51cd84
I20260923 23:13:48.677644 3952 raft_consensus.cc:2454] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c2c28dd7fa49aab00122340e51cd84 in term 1.
I20260923 23:13:48.677644 3684 raft_consensus.cc:2454] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b8c2c28dd7fa49aab00122340e51cd84 in term 1.
I20260923 23:13:48.677764 4018 raft_consensus.cc:2726] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260923 23:13:48.677803 3751 leader_election.cc:302] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 46206ef3aa5e4db09619d7d242f76c57, b8c2c28dd7fa49aab00122340e51cd84; no voters:
I20260923 23:13:48.677894 4020 raft_consensus.cc:2781] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:48.678032 4020 raft_consensus.cc:686] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 LEADER]: Becoming Leader. State: Replica: b8c2c28dd7fa49aab00122340e51cd84, State: Running, Role: LEADER
I20260923 23:13:48.678102 4020 consensus_queue.cc:237] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } }
I20260923 23:13:48.678838 3539 catalog_manager.cc:5495] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 reported cstate change: term changed from 0 to 1, leader changed from <none> to b8c2c28dd7fa49aab00122340e51cd84 (127.16.122.130). New cstate: current_term: 1 leader_uuid: "b8c2c28dd7fa49aab00122340e51cd84" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b8c2c28dd7fa49aab00122340e51cd84" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 40121 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 } health_report { overall_health: UNKNOWN } } }
I20260923 23:13:48.691794 4009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.689615 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54326 (local address 127.16.122.130:40121)
0923 23:13:48.689665 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.689667 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.689676 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:48.689712 (+ 36us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.689713 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.689732 (+ 19us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.689786 (+ 54us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:48.689913 (+ 127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.690381 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.691120 (+ 739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.691312 (+ 192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.691373 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.691483 (+ 110us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:48.691586 (+ 103us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:48.691660 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.691674 (+ 14us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.691696 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:13:48.693557 3952 raft_consensus.cc:1261] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 1 FOLLOWER]: Refusing update from remote peer b8c2c28dd7fa49aab00122340e51cd84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:48.693841 3684 raft_consensus.cc:1261] T 481d8be940604685ab931e58dda9048c P 46206ef3aa5e4db09619d7d242f76c57 [term 1 FOLLOWER]: Refusing update from remote peer b8c2c28dd7fa49aab00122340e51cd84: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:48.693832 4020 consensus_queue.cc:1035] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d5c7e1ed5abe453d865b45f732ef6592" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 35419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:48.694105 4020 consensus_queue.cc:1035] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [LEADER]: Connected to new peer: Peer: permanent_uuid: "46206ef3aa5e4db09619d7d242f76c57" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 36027 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:48.696290 4028 mvcc.cc:205] Tried to move back new op lower bound from 7332680616726482944 to 7332680616666202112. Current Snapshot: MvccSnapshot[applied={T|T < 7332680616726482944}]
I20260923 23:13:48.696655 4029 mvcc.cc:205] Tried to move back new op lower bound from 7332680616726482944 to 7332680616666202112. Current Snapshot: MvccSnapshot[applied={T|T < 7332680616726482944}]
I20260923 23:13:48.698905 3538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:48.699002 3538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:43906:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:48.699751 3538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 0caaba265040443c897a490b77786936: Sending DeleteTablet for 3 replicas of tablet 481d8be940604685ab931e58dda9048c
I20260923 23:13:48.700029 3664 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d8be940604685ab931e58dda9048c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:48 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:42792
I20260923 23:13:48.700052 3798 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d8be940604685ab931e58dda9048c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:48 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:54314
I20260923 23:13:48.700137 3932 tablet_service.cc:1514] Processing DeleteTablet for tablet 481d8be940604685ab931e58dda9048c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:48 UTC) from {username='kudu', principal='kudu/127.16.122.190@KRBTEST.COM'} at 127.0.0.1:58506
I20260923 23:13:48.700416 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3601
I20260923 23:13:48.700435 4039 tablet_replica.cc:330] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: stopping tablet replica
I20260923 23:13:48.700563 4039 raft_consensus.cc:2229] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:48.700708 4039 raft_consensus.cc:2258] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:48.701843 4039 ts_tablet_manager.cc:1905] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:48.702466 4038 tablet_replica.cc:330] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: stopping tablet replica
I20260923 23:13:48.702570 4038 raft_consensus.cc:2229] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:48.702687 4038 raft_consensus.cc:2258] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:48.703132 4038 ts_tablet_manager.cc:1905] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:48.703998 4039 ts_tablet_manager.cc:1918] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:48.704108 4039 log.cc:1196] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/481d8be940604685ab931e58dda9048c
I20260923 23:13:48.704407 4039 ts_tablet_manager.cc:1939] T 481d8be940604685ab931e58dda9048c P d5c7e1ed5abe453d865b45f732ef6592: Deleting consensus metadata
I20260923 23:13:48.704813 4038 ts_tablet_manager.cc:1918] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:48.704903 4038 log.cc:1196] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/481d8be940604685ab931e58dda9048c
I20260923 23:13:48.705175 4038 ts_tablet_manager.cc:1939] T 481d8be940604685ab931e58dda9048c P b8c2c28dd7fa49aab00122340e51cd84: Deleting consensus metadata
I20260923 23:13:48.705281 3523 catalog_manager.cc:4841] TS d5c7e1ed5abe453d865b45f732ef6592 (127.16.122.131:35419): tablet 481d8be940604685ab931e58dda9048c (table test-table [id=a7a6306cf5524f47afe7ba239eae7076]) successfully deleted
I20260923 23:13:48.705529 3526 catalog_manager.cc:4841] TS b8c2c28dd7fa49aab00122340e51cd84 (127.16.122.130:40121): tablet 481d8be940604685ab931e58dda9048c (table test-table [id=a7a6306cf5524f47afe7ba239eae7076]) successfully deleted
W20260923 23:13:48.706349 3523 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:36027 (error 108)
I20260923 23:13:48.706609 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3735
I20260923 23:13:48.707181 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.706727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:36027 (local address 127.0.0.1:42796)
0923 23:13:48.707103 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:48.707145 (+ 42us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:36027: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":357}
W20260923 23:13:48.707294 3523 catalog_manager.cc:4584] TS 46206ef3aa5e4db09619d7d242f76c57 (127.16.122.129:36027): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 481d8be940604685ab931e58dda9048c: Network error: Client connection negotiation failed: client connection to 127.16.122.129:36027: connect: Connection refused (error 111)
I20260923 23:13:48.711127 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3869
I20260923 23:13:48.715165 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 3507
2026-09-23T23:13:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (1774 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-09-23T23:13:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:48Z Disabled control of system clock
I20260923 23:13:48.739900 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40469
--webserver_interface=127.16.122.190
--webserver_port=0
--builtin_ntp_servers=127.16.122.148:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40469
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:48.817819 4044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.818034 4044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.818053 4044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.819150 4044 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:48.819185 4044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.819196 4044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.819207 4044 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:48.819217 4044 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:48.820478 4044 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40469
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40469
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4044
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.820669 4044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.820914 4044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:48.825487 4044 server_base.cc:1034] running on GCE node
W20260923 23:13:48.825459 4049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.825448 4052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.825459 4050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.825848 4044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.826105 4044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.827236 4044 hybrid_clock.cc:648] HybridClock initialized: now 1790205228827225 us; error 29 us; skew 500 ppm
I20260923 23:13:48.828352 4044 webserver.cc:460] Webserver started at http://127.16.122.190:35229/ using document root <none> and password file <none>
I20260923 23:13:48.828536 4044 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.828574 4044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.828665 4044 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.829643 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "9199c1f5c609422f8d44385bb45dec7b"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "1c980674eb3d0df261d848e0b42918f4"
server_key_iv: "bc796409b8b8aad188f7b90a5aa038e8"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.829931 4044 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "9199c1f5c609422f8d44385bb45dec7b"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "1c980674eb3d0df261d848e0b42918f4"
server_key_iv: "bc796409b8b8aad188f7b90a5aa038e8"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.831058 4044 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:48.831684 4058 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.831844 4044 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:48.831924 4044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "9199c1f5c609422f8d44385bb45dec7b"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "1c980674eb3d0df261d848e0b42918f4"
server_key_iv: "bc796409b8b8aad188f7b90a5aa038e8"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.831977 4044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.844760 4044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.845062 4044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.845165 4044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.848704 4044 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:40469
I20260923 23:13:48.848739 4110 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:40469 every 8 connection(s)
I20260923 23:13:48.849059 4044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:48.849941 4111 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:48.852186 4111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: Bootstrap starting.
I20260923 23:13:48.852715 4111 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:48.852977 4111 log.cc:824] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:48.853220 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4044
I20260923 23:13:48.853307 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:48.853461 16874 external_mini_cluster.cc:1428] Setting key 36b22c5ec11727d84bf262ca9e0332de
I20260923 23:13:48.853614 4111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: No bootstrap required, opened a new log
I20260923 23:13:48.855460 4111 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } }
I20260923 23:13:48.855593 4111 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:48.855614 4111 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9199c1f5c609422f8d44385bb45dec7b, State: Initialized, Role: FOLLOWER
I20260923 23:13:48.855717 4111 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } }
I20260923 23:13:48.855808 4111 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:48.855831 4111 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:48.855876 4111 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:48.856380 4111 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } }
I20260923 23:13:48.856472 4111 leader_election.cc:302] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9199c1f5c609422f8d44385bb45dec7b; no voters:
I20260923 23:13:48.856633 4111 leader_election.cc:288] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:48.856663 4116 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:48.856777 4116 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [term 1 LEADER]: Becoming Leader. State: Replica: 9199c1f5c609422f8d44385bb45dec7b, State: Running, Role: LEADER
I20260923 23:13:48.856863 4116 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } }
I20260923 23:13:48.856892 4111 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:48.857191 4116 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9199c1f5c609422f8d44385bb45dec7b. Latest consensus state: current_term: 1 leader_uuid: "9199c1f5c609422f8d44385bb45dec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } } }
I20260923 23:13:48.857200 4118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9199c1f5c609422f8d44385bb45dec7b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9199c1f5c609422f8d44385bb45dec7b" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40469 } } }
I20260923 23:13:48.857277 4116 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:48.857291 4118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b [sys.catalog]: This master's current role is: LEADER
W20260923 23:13:48.858568 4132 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:13:48.858644 4132 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:13:48.858707 4122 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:48.858855 4122 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:48.859699 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:48.853942 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33952 (local address 127.16.122.190:40469)
0923 23:13:48.854138 (+ 196us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:48.854145 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:48.854168 (+ 23us) server_negotiation.cc:407] Connection header received
0923 23:13:48.854766 (+ 598us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:48.854772 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:48.854831 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:48.854952 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:48.855184 (+ 232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.856527 (+ 1343us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.857535 (+ 1008us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:48.858179 (+ 644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:48.858319 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:48.859126 (+ 807us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:48.859129 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:48.859130 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:48.859298 (+ 168us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:48.859332 (+ 34us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:48.859343 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:48.859397 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:48.859467 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:13:48.859545 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":120,"thread_start_us":75,"threads_started":1}
I20260923 23:13:48.860445 4122 catalog_manager.cc:1347] Generated new cluster ID: 4ccd12d957fb4490a51b2659a9ef1e5d
I20260923 23:13:48.860487 4122 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:48.870633 4122 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:48.871188 4122 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:48.876852 4122 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: Generated new TSK 0
I20260923 23:13:48.877030 4122 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:48.884447 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.129:0
--local_ip_for_outbound_sockets=127.16.122.129
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--builtin_ntp_servers=127.16.122.148:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260923 23:13:48.974646 4135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:48.974845 4135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:48.974865 4135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:48.976047 4135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:48.976089 4135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:48.976115 4135 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.129
I20260923 23:13:48.977432 4135 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.122.129
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4135
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.129
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:48.977663 4135 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:48.977942 4135 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:48.982685 4140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:48.982818 4141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.982751 4135 server_base.cc:1034] running on GCE node
W20260923 23:13:48.982713 4143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:48.983090 4135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:48.983330 4135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:48.984479 4135 hybrid_clock.cc:648] HybridClock initialized: now 1790205228984465 us; error 31 us; skew 500 ppm
I20260923 23:13:48.985677 4135 webserver.cc:460] Webserver started at http://127.16.122.129:45821/ using document root <none> and password file <none>
I20260923 23:13:48.985922 4135 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:48.985965 4135 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:48.986083 4135 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:48.987181 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data/instance:
uuid: "39c2969141404a0aa7b179655cafe315"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "aad2621a73fefd8fddab07b107274f03"
server_key_iv: "d6450f6dbeab4efae96ab5747032aea9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.987491 4135 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance:
uuid: "39c2969141404a0aa7b179655cafe315"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "aad2621a73fefd8fddab07b107274f03"
server_key_iv: "d6450f6dbeab4efae96ab5747032aea9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.988670 4135 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:13:48.989353 4149 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.989490 4135 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.989564 4135 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal
uuid: "39c2969141404a0aa7b179655cafe315"
format_stamp: "Formatted at 2026-09-23 23:13:48 on dist-test-slave-95hh"
server_key: "aad2621a73fefd8fddab07b107274f03"
server_key_iv: "d6450f6dbeab4efae96ab5747032aea9"
server_key_version: "encryptionkey@0"
I20260923 23:13:48.989634 4135 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:48.997388 4135 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:48.997658 4135 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:48.997790 4135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:48.997983 4135 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:48.998286 4135 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:48.998318 4135 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:48.998345 4135 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:48.998368 4135 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.003796 4135 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.129:43357
I20260923 23:13:49.003834 4262 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.129:43357 every 8 connection(s)
I20260923 23:13:49.004158 4135 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/data/info.pb
I20260923 23:13:49.009192 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.004846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.129:47293 (local address 127.16.122.190:40469)
0923 23:13:49.004902 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.004904 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.005133 (+ 229us) server_negotiation.cc:407] Connection header received
0923 23:13:49.005654 (+ 521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.005655 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.005671 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.005734 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:49.006419 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.006921 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.007917 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.008105 (+ 188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.008159 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.008940 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:49.008942 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:49.008942 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:49.008947 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:49.008963 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:49.008967 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:49.009008 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.009082 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.009111 (+ 29us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33}
I20260923 23:13:49.009392 4265 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.004838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40469 (local address 127.16.122.129:47293)
0923 23:13:49.005016 (+ 178us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.005050 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.005554 (+ 504us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.005812 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.005817 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.005825 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:49.006328 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.006332 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.007006 (+ 674us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.007007 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.007836 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.007839 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.008173 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.008174 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.008309 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.008807 (+ 498us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:49.008807 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:49.008836 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:49.008862 (+ 26us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:49.009015 (+ 153us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:49.009020 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:49.009023 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.009083 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.009186 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":76,"threads_started":1}
I20260923 23:13:49.009625 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4135
I20260923 23:13:49.009727 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-0/wal/instance
I20260923 23:13:49.009837 4263 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40469
I20260923 23:13:49.009927 4263 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:49.009933 16874 external_mini_cluster.cc:1428] Setting key 80f8483059d4d7a5f7812d9b2d0d6529
I20260923 23:13:49.010115 4263 heartbeater.cc:511] Master 127.16.122.190:40469 requested a full tablet report, sending...
I20260923 23:13:49.010603 4074 ts_manager.cc:194] Registered new tserver with Master: 39c2969141404a0aa7b179655cafe315 (127.16.122.129:43357)
I20260923 23:13:49.010849 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.130:0
--local_ip_for_outbound_sockets=127.16.122.130
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--builtin_ntp_servers=127.16.122.148:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:49.011451 4074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.129:47293
W20260923 23:13:49.088236 4266 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:49.088426 4266 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:49.088445 4266 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:49.089486 4266 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:49.089515 4266 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:49.089537 4266 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.130
I20260923 23:13:49.090785 4266 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.122.130
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4266
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.130
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:49.091006 4266 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:49.091339 4266 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:49.096555 4266 server_base.cc:1034] running on GCE node
W20260923 23:13:49.096486 4274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:49.096491 4272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:49.096514 4271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:49.096901 4266 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:49.097132 4266 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:49.098270 4266 hybrid_clock.cc:648] HybridClock initialized: now 1790205229098259 us; error 29 us; skew 500 ppm
I20260923 23:13:49.099483 4266 webserver.cc:460] Webserver started at http://127.16.122.130:44449/ using document root <none> and password file <none>
I20260923 23:13:49.099720 4266 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:49.099764 4266 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:49.099901 4266 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:49.100960 4266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data/instance:
uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "8566d49d3b997ec91a8993ceb6b4592d"
server_key_iv: "c1e7ee08120489b88f0cc88bdf671c64"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.101275 4266 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance:
uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "8566d49d3b997ec91a8993ceb6b4592d"
server_key_iv: "c1e7ee08120489b88f0cc88bdf671c64"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.102480 4266 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:49.103114 4280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.103255 4266 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.103335 4266 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal
uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "8566d49d3b997ec91a8993ceb6b4592d"
server_key_iv: "c1e7ee08120489b88f0cc88bdf671c64"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.103401 4266 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:49.115132 4266 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:49.115412 4266 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:49.115538 4266 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:49.115741 4266 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:49.116065 4266 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:49.116097 4266 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.116125 4266 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:49.116147 4266 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.121402 4266 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.130:38765
I20260923 23:13:49.121454 4393 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.130:38765 every 8 connection(s)
I20260923 23:13:49.121757 4266 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/data/info.pb
I20260923 23:13:49.125072 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4266
I20260923 23:13:49.125166 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal/instance
I20260923 23:13:49.125370 16874 external_mini_cluster.cc:1428] Setting key af4cfeb711b354e330a3b9e49c9e7307
I20260923 23:13:49.126477 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.122.131:0
--local_ip_for_outbound_sockets=127.16.122.131
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--builtin_ntp_servers=127.16.122.148:40213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260923 23:13:49.127280 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.122407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:59807 (local address 127.16.122.190:40469)
0923 23:13:49.122470 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.122472 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.122681 (+ 209us) server_negotiation.cc:407] Connection header received
0923 23:13:49.123200 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.123202 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.123213 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.123262 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:49.123899 (+ 637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.124375 (+ 476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.125334 (+ 959us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.125592 (+ 258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.125662 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.126501 (+ 839us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:49.126502 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:49.126502 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:49.126507 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:49.126524 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:49.126528 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:49.126585 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.127171 (+ 586us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.127201 (+ 30us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20260923 23:13:49.127642 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.122403 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40469 (local address 127.16.122.130:59807)
0923 23:13:49.122575 (+ 172us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.122609 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.123115 (+ 506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.123343 (+ 228us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.123347 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.123355 (+ 8us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:49.123836 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.123841 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.124433 (+ 592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.124434 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.125241 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.125244 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.125671 (+ 427us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.125672 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.125823 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.126366 (+ 543us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:49.126366 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:49.126395 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:49.126418 (+ 23us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:49.127086 (+ 668us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:49.127091 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:49.127094 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.127162 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.127266 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"thread_start_us":81,"threads_started":1}
I20260923 23:13:49.128221 4394 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40469
I20260923 23:13:49.128312 4394 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:49.128484 4394 heartbeater.cc:511] Master 127.16.122.190:40469 requested a full tablet report, sending...
I20260923 23:13:49.128847 4074 ts_manager.cc:194] Registered new tserver with Master: 38bb5bc5e9cf42ff8b9c00ae3b080d0b (127.16.122.130:38765)
I20260923 23:13:49.129194 4074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.130:59807
W20260923 23:13:49.205686 4397 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:49.205886 4397 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:49.205906 4397 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:49.207110 4397 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:49.207151 4397 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:49.207175 4397 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.122.131
I20260923 23:13:49.208514 4397 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.122.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.122.131
--webserver_port=0
--tserver_master_addrs=127.16.122.190:40469
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4397
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.122.131
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:49.208736 4397 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:49.209007 4397 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:49.213984 4403 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:49.213990 4402 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:49.214054 4405 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:49.214118 4397 server_base.cc:1034] running on GCE node
I20260923 23:13:49.214402 4397 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:49.214637 4397 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:49.215790 4397 hybrid_clock.cc:648] HybridClock initialized: now 1790205229215775 us; error 28 us; skew 500 ppm
I20260923 23:13:49.216948 4397 webserver.cc:460] Webserver started at http://127.16.122.131:35311/ using document root <none> and password file <none>
I20260923 23:13:49.217188 4397 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:49.217239 4397 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:49.217361 4397 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:49.218379 4397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data/instance:
uuid: "58dbc937fd8e4d19866a7178b6362fde"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "3a8f1ec83f6202e26a65d4806603b512"
server_key_iv: "2c8888db691e6456d6a14aaf20dbc5cc"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.218703 4397 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance:
uuid: "58dbc937fd8e4d19866a7178b6362fde"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "3a8f1ec83f6202e26a65d4806603b512"
server_key_iv: "2c8888db691e6456d6a14aaf20dbc5cc"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.219926 4397 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:49.220594 4411 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.220734 4397 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:49.220814 4397 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal
uuid: "58dbc937fd8e4d19866a7178b6362fde"
format_stamp: "Formatted at 2026-09-23 23:13:49 on dist-test-slave-95hh"
server_key: "3a8f1ec83f6202e26a65d4806603b512"
server_key_iv: "2c8888db691e6456d6a14aaf20dbc5cc"
server_key_version: "encryptionkey@0"
I20260923 23:13:49.220873 4397 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:49.244537 4397 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:49.244870 4397 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:49.245003 4397 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:49.245229 4397 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260923 23:13:49.245563 4397 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260923 23:13:49.245596 4397 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.245625 4397 ts_tablet_manager.cc:610] Registered 0 tablets
I20260923 23:13:49.245642 4397 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:49.250815 4397 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.131:39105
I20260923 23:13:49.250882 4524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.131:39105 every 8 connection(s)
I20260923 23:13:49.251212 4397 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/data/info.pb
I20260923 23:13:49.251648 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4397
I20260923 23:13:49.251734 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal/instance
I20260923 23:13:49.251880 16874 external_mini_cluster.cc:1428] Setting key 10a534e2154828c8404ffeaa4c299f38
I20260923 23:13:49.257092 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.252363 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.131:33601 (local address 127.16.122.190:40469)
0923 23:13:49.252434 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.252437 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.252725 (+ 288us) server_negotiation.cc:407] Connection header received
0923 23:13:49.253336 (+ 611us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.253338 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.253351 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.253404 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:49.254150 (+ 746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.254666 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.255756 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.255958 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.256022 (+ 64us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.256788 (+ 766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:49.256790 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:49.256790 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:49.256796 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:49.256814 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:49.256819 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:49.256870 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.256950 (+ 80us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.256981 (+ 31us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:13:49.257323 4527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.252364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40469 (local address 127.16.122.131:33601)
0923 23:13:49.252584 (+ 220us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.252632 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.253237 (+ 605us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.253481 (+ 244us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.253485 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.253504 (+ 19us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:13:49.254029 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.254035 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.254761 (+ 726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.254763 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.255653 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.255656 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.256036 (+ 380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.256037 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.256178 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.256652 (+ 474us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0923 23:13:49.256652 (+ 0us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:13:49.256681 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0923 23:13:49.256705 (+ 24us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:13:49.256876 (+ 171us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:13:49.256881 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:13:49.256884 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.256950 (+ 66us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.257123 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":141,"thread_start_us":88,"threads_started":1}
I20260923 23:13:49.257822 4525 heartbeater.cc:347] Connected to a master server at 127.16.122.190:40469
I20260923 23:13:49.257932 4525 heartbeater.cc:464] Registering TS with master...
I20260923 23:13:49.258150 4525 heartbeater.cc:511] Master 127.16.122.190:40469 requested a full tablet report, sending...
I20260923 23:13:49.258502 4074 ts_manager.cc:194] Registered new tserver with Master: 58dbc937fd8e4d19866a7178b6362fde (127.16.122.131:39105)
I20260923 23:13:49.258860 4074 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.122.131:33601
I20260923 23:13:49.263703 16874 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260923 23:13:49.267491 4115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.265032 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33956 (local address 127.16.122.190:40469)
0923 23:13:49.265073 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.265075 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.265193 (+ 118us) server_negotiation.cc:407] Connection header received
0923 23:13:49.265236 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.265236 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.265245 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.265288 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:49.265394 (+ 106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.265859 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.266520 (+ 661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.266689 (+ 169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.266729 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.267305 (+ 576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:49.267306 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:49.267307 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:49.267311 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:49.267324 (+ 13us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:49.267328 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:49.267364 (+ 36us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.267394 (+ 30us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.267415 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:13:49.268754 4074 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:33956:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260923 23:13:49.270239 4074 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260923 23:13:49.276042 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.130:38765 (local address 127.0.0.1:53276)
0923 23:13:49.272468 (+ 125us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.272476 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.272513 (+ 37us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.272856 (+ 343us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.272856 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.272859 (+ 3us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:49.272902 (+ 43us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.272904 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.273779 (+ 875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.273779 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.274914 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.274917 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.275759 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.275760 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.275830 (+ 70us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.275846 (+ 16us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.275915 (+ 69us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.275968 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111,"thread_start_us":81,"threads_started":1}
I20260923 23:13:49.276052 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53276 (local address 127.16.122.130:38765)
0923 23:13:49.272722 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.272725 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.272734 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:49.272768 (+ 34us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.272770 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.272782 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.272849 (+ 67us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:49.272978 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.273703 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.274985 (+ 1282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.275678 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.275765 (+ 87us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.275859 (+ 94us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.275930 (+ 71us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.275971 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":136,"thread_start_us":89,"threads_started":1}
I20260923 23:13:49.276242 4533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272159 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43357 (local address 127.0.0.1:55388)
0923 23:13:49.272436 (+ 277us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.272446 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.272498 (+ 52us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.272753 (+ 255us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.272765 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.272767 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:49.272831 (+ 64us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.272833 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.274164 (+ 1331us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.274165 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.275206 (+ 1041us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.275208 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.276053 (+ 845us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.276055 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.276113 (+ 58us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.276118 (+ 5us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.276146 (+ 28us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.276185 (+ 39us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":264,"thread_start_us":96,"threads_started":1}
I20260923 23:13:49.276476 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55388 (local address 127.16.122.129:43357)
0923 23:13:49.272600 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.272606 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.272615 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:49.272655 (+ 40us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.272657 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.272672 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.272747 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:49.272958 (+ 211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.274083 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.275284 (+ 1201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.275965 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.276305 (+ 340us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.276371 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.276390 (+ 19us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.276417 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":85,"threads_started":1}
I20260923 23:13:49.276911 4534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272145 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39105 (local address 127.0.0.1:56598)
0923 23:13:49.272309 (+ 164us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.272350 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.272408 (+ 58us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.272902 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.272902 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.272904 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:49.272934 (+ 30us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.272936 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.275073 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.275074 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.275822 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.275824 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.276690 (+ 866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.276691 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.276751 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.276763 (+ 12us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.276814 (+ 51us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.276849 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":132,"thread_start_us":57,"threads_started":1}
I20260923 23:13:49.276948 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.272322 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56598 (local address 127.16.122.131:39105)
0923 23:13:49.272610 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.272614 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.272621 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:49.272658 (+ 37us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.272660 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.272675 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.272752 (+ 77us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:49.273008 (+ 256us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.274083 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.275911 (+ 1828us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.276619 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.276695 (+ 76us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.276778 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.276830 (+ 52us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.276872 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":255,"thread_start_us":69,"threads_started":1}
I20260923 23:13:49.277125 4328 tablet_service.cc:1467] Processing CreateTablet for tablet 930082e074384d88b1df9a98abe4326b (DEFAULT_TABLE table=test-table [id=68582f2d144c4a49a974b75e3b87a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:49.277127 4197 tablet_service.cc:1467] Processing CreateTablet for tablet 930082e074384d88b1df9a98abe4326b (DEFAULT_TABLE table=test-table [id=68582f2d144c4a49a974b75e3b87a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:49.277442 4328 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930082e074384d88b1df9a98abe4326b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:49.277447 4197 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930082e074384d88b1df9a98abe4326b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:49.279740 4459 tablet_service.cc:1467] Processing CreateTablet for tablet 930082e074384d88b1df9a98abe4326b (DEFAULT_TABLE table=test-table [id=68582f2d144c4a49a974b75e3b87a118]), partition=RANGE (key) PARTITION UNBOUNDED
I20260923 23:13:49.280074 4459 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 930082e074384d88b1df9a98abe4326b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:49.282298 4541 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Bootstrap starting.
I20260923 23:13:49.282341 4539 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Bootstrap starting.
I20260923 23:13:49.282472 4540 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: Bootstrap starting.
I20260923 23:13:49.282841 4541 tablet_bootstrap.cc:654] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:49.282958 4539 tablet_bootstrap.cc:654] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:49.283010 4540 tablet_bootstrap.cc:654] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:49.283144 4541 log.cc:824] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:49.283303 4539 log.cc:824] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:49.283326 4540 log.cc:824] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:49.283978 4541 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: No bootstrap required, opened a new log
I20260923 23:13:49.283994 4540 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: No bootstrap required, opened a new log
I20260923 23:13:49.284049 4541 ts_tablet_manager.cc:1397] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:49.284049 4540 ts_tablet_manager.cc:1397] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: Time spent bootstrapping tablet: real 0.002s user 0.001s sys 0.000s
I20260923 23:13:49.284142 4539 tablet_bootstrap.cc:492] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: No bootstrap required, opened a new log
I20260923 23:13:49.284204 4539 ts_tablet_manager.cc:1397] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Time spent bootstrapping tablet: real 0.002s user 0.000s sys 0.001s
I20260923 23:13:49.285930 4539 raft_consensus.cc:348] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286054 4539 raft_consensus.cc:374] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:49.286084 4539 raft_consensus.cc:729] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38bb5bc5e9cf42ff8b9c00ae3b080d0b, State: Initialized, Role: FOLLOWER
I20260923 23:13:49.286192 4539 consensus_queue.cc:260] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286444 4539 ts_tablet_manager.cc:1428] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260923 23:13:49.286451 4394 heartbeater.cc:503] Master 127.16.122.190:40469 was elected leader, sending a full tablet report...
I20260923 23:13:49.286468 4540 raft_consensus.cc:348] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286468 4541 raft_consensus.cc:348] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286610 4540 raft_consensus.cc:374] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:49.286610 4541 raft_consensus.cc:374] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:49.286643 4540 raft_consensus.cc:729] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c2969141404a0aa7b179655cafe315, State: Initialized, Role: FOLLOWER
I20260923 23:13:49.286643 4541 raft_consensus.cc:729] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 58dbc937fd8e4d19866a7178b6362fde, State: Initialized, Role: FOLLOWER
I20260923 23:13:49.286746 4540 consensus_queue.cc:260] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286765 4541 consensus_queue.cc:260] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.286990 4540 ts_tablet_manager.cc:1428] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:49.286990 4541 ts_tablet_manager.cc:1428] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260923 23:13:49.287114 4525 heartbeater.cc:503] Master 127.16.122.190:40469 was elected leader, sending a full tablet report...
I20260923 23:13:49.287134 4263 heartbeater.cc:503] Master 127.16.122.190:40469 was elected leader, sending a full tablet report...
W20260923 23:13:49.372290 4395 tablet.cc:2369] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260923 23:13:49.387722 4545 raft_consensus.cc:482] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260923 23:13:49.387853 4545 raft_consensus.cc:504] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.388278 4545 leader_election.cc:288] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 58dbc937fd8e4d19866a7178b6362fde (127.16.122.131:39105), 39c2969141404a0aa7b179655cafe315 (127.16.122.129:43357)
I20260923 23:13:49.390826 4535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.388535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:35189 (local address 127.16.122.129:43357)
0923 23:13:49.388595 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.388597 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.388604 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:49.388630 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.388631 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.388640 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.388684 (+ 44us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:49.388834 (+ 150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.389296 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390181 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.390528 (+ 347us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390588 (+ 60us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.390596 (+ 8us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.390718 (+ 122us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.390763 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:13:49.390827 4396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.388416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43357 (local address 127.16.122.130:35189)
0923 23:13:49.388460 (+ 44us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.388472 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.388511 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.388701 (+ 190us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.388702 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.388704 (+ 2us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:49.388769 (+ 65us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.388771 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.389370 (+ 599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.389371 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.390091 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.390093 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.390597 (+ 504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390598 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.390658 (+ 60us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.390661 (+ 3us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.390705 (+ 44us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.390750 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":24}
I20260923 23:13:49.391135 4536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.388525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.122.130:38287 (local address 127.16.122.131:39105)
0923 23:13:49.388573 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.388575 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.388721 (+ 146us) server_negotiation.cc:407] Connection header received
0923 23:13:49.388801 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.388802 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.388812 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.388857 (+ 45us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0923 23:13:49.389018 (+ 161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.389479 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390388 (+ 909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.390740 (+ 352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390807 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.390814 (+ 7us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.391009 (+ 195us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.391073 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25}
I20260923 23:13:49.391144 4548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.388417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.131:39105 (local address 127.16.122.130:38287)
0923 23:13:49.388641 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.388651 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:13:49.388728 (+ 77us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:13:49.388878 (+ 150us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:13:49.388879 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:13:49.388880 (+ 1us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0923 23:13:49.388951 (+ 71us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.388953 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.389558 (+ 605us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.389560 (+ 2us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.390302 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:13:49.390304 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.390814 (+ 510us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.390815 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:13:49.390884 (+ 69us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.390888 (+ 4us) client_negotiation.cc:769] Sending connection context
0923 23:13:49.390979 (+ 91us) client_negotiation.cc:240] Negotiation successful
0923 23:13:49.391021 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":208,"mutex_wait_us":45,"thread_start_us":115,"threads_started":1}
I20260923 23:13:49.391165 4217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930082e074384d88b1df9a98abe4326b" candidate_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2969141404a0aa7b179655cafe315" is_pre_election: true
I20260923 23:13:49.391319 4217 raft_consensus.cc:2454] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bb5bc5e9cf42ff8b9c00ae3b080d0b in term 0.
I20260923 23:13:49.391492 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930082e074384d88b1df9a98abe4326b" candidate_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dbc937fd8e4d19866a7178b6362fde" is_pre_election: true
I20260923 23:13:49.391541 4284 leader_election.cc:302] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bb5bc5e9cf42ff8b9c00ae3b080d0b, 39c2969141404a0aa7b179655cafe315; no voters:
I20260923 23:13:49.391664 4479 raft_consensus.cc:2454] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38bb5bc5e9cf42ff8b9c00ae3b080d0b in term 0.
I20260923 23:13:49.391702 4545 raft_consensus.cc:2781] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260923 23:13:49.391774 4545 raft_consensus.cc:482] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260923 23:13:49.391801 4545 raft_consensus.cc:3037] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:49.392524 4545 raft_consensus.cc:504] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.392659 4545 leader_election.cc:288] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [CANDIDATE]: Term 1 election: Requested vote from peers 58dbc937fd8e4d19866a7178b6362fde (127.16.122.131:39105), 39c2969141404a0aa7b179655cafe315 (127.16.122.129:43357)
I20260923 23:13:49.392807 4217 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930082e074384d88b1df9a98abe4326b" candidate_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39c2969141404a0aa7b179655cafe315"
I20260923 23:13:49.392822 4479 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "930082e074384d88b1df9a98abe4326b" candidate_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "58dbc937fd8e4d19866a7178b6362fde"
I20260923 23:13:49.392875 4217 raft_consensus.cc:3037] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:49.392889 4479 raft_consensus.cc:3037] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:49.393602 4217 raft_consensus.cc:2454] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bb5bc5e9cf42ff8b9c00ae3b080d0b in term 1.
I20260923 23:13:49.393726 4479 raft_consensus.cc:2454] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38bb5bc5e9cf42ff8b9c00ae3b080d0b in term 1.
I20260923 23:13:49.393798 4284 leader_election.cc:302] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38bb5bc5e9cf42ff8b9c00ae3b080d0b, 39c2969141404a0aa7b179655cafe315; no voters:
I20260923 23:13:49.393936 4545 raft_consensus.cc:2781] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:49.394111 4545 raft_consensus.cc:686] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 1 LEADER]: Becoming Leader. State: Replica: 38bb5bc5e9cf42ff8b9c00ae3b080d0b, State: Running, Role: LEADER
I20260923 23:13:49.394200 4545 consensus_queue.cc:237] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } }
I20260923 23:13:49.395114 4074 catalog_manager.cc:5495] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b reported cstate change: term changed from 0 to 1, leader changed from <none> to 38bb5bc5e9cf42ff8b9c00ae3b080d0b (127.16.122.130). New cstate: current_term: 1 leader_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "38bb5bc5e9cf42ff8b9c00ae3b080d0b" member_type: VOTER last_known_addr { host: "127.16.122.130" port: 38765 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 } health_report { overall_health: UNKNOWN } } }
I20260923 23:13:49.418990 4538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.416921 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53288 (local address 127.16.122.130:38765)
0923 23:13:49.416967 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:49.416969 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:49.416978 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:13:49.417022 (+ 44us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:49.417022 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:49.417040 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:49.417100 (+ 60us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:13:49.417235 (+ 135us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.417699 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.418402 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:49.418575 (+ 173us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:49.418615 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:49.418741 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:13:49.418836 (+ 95us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:13:49.418890 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:49.418902 (+ 12us) server_negotiation.cc:299] Negotiation successful
0923 23:13:49.418915 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:13:49.420627 4217 raft_consensus.cc:1261] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 1 FOLLOWER]: Refusing update from remote peer 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:49.420625 4479 raft_consensus.cc:1261] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 1 FOLLOWER]: Refusing update from remote peer 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:13:49.420951 4549 consensus_queue.cc:1035] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "58dbc937fd8e4d19866a7178b6362fde" member_type: VOTER last_known_addr { host: "127.16.122.131" port: 39105 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:49.421061 4545 consensus_queue.cc:1035] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [LEADER]: Connected to new peer: Peer: permanent_uuid: "39c2969141404a0aa7b179655cafe315" member_type: VOTER last_known_addr { host: "127.16.122.129" port: 43357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:13:49.423630 4553 mvcc.cc:205] Tried to move back new op lower bound from 7332680619704639488 to 7332680619599441920. Current Snapshot: MvccSnapshot[applied={T|T < 7332680619704639488}]
I20260923 23:13:49.424081 4554 mvcc.cc:205] Tried to move back new op lower bound from 7332680619704639488 to 7332680619599441920. Current Snapshot: MvccSnapshot[applied={T|T < 7332680619704639488}]
I20260923 23:13:49.426079 4073 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:49.426172 4073 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:33956:
table { table_name: "test-table" } modify_external_catalogs: true
I20260923 23:13:49.426887 4073 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9199c1f5c609422f8d44385bb45dec7b: Sending DeleteTablet for 3 replicas of tablet 930082e074384d88b1df9a98abe4326b
I20260923 23:13:49.427256 4459 tablet_service.cc:1514] Processing DeleteTablet for tablet 930082e074384d88b1df9a98abe4326b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:49 UTC) from {username='slave'} at 127.0.0.1:56598
I20260923 23:13:49.427263 4197 tablet_service.cc:1514] Processing DeleteTablet for tablet 930082e074384d88b1df9a98abe4326b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:49 UTC) from {username='slave'} at 127.0.0.1:55388
I20260923 23:13:49.427461 4562 tablet_replica.cc:330] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315: stopping tablet replica
I20260923 23:13:49.427511 4328 tablet_service.cc:1514] Processing DeleteTablet for tablet 930082e074384d88b1df9a98abe4326b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-23 23:13:49 UTC) from {username='slave'} at 127.0.0.1:53276
I20260923 23:13:49.427553 4562 raft_consensus.cc:2229] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:49.427647 4562 raft_consensus.cc:2258] T 930082e074384d88b1df9a98abe4326b P 39c2969141404a0aa7b179655cafe315 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:49.427711 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4135
I20260923 23:13:49.429039 4563 tablet_replica.cc:330] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: stopping tablet replica
I20260923 23:13:49.429137 4563 raft_consensus.cc:2229] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 1 FOLLOWER]: Raft consensus shutting down.
I20260923 23:13:49.429250 4563 raft_consensus.cc:2258] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:49.429394 4564 tablet_replica.cc:330] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: stopping tablet replica
I20260923 23:13:49.429481 4564 raft_consensus.cc:2229] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 1 LEADER]: Raft consensus shutting down.
I20260923 23:13:49.429589 4564 raft_consensus.cc:2258] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260923 23:13:49.429929 4563 ts_tablet_manager.cc:1905] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260923 23:13:49.431375 4563 ts_tablet_manager.cc:1918] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:49.431437 4563 log.cc:1196] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-2/wal/wals/930082e074384d88b1df9a98abe4326b
I20260923 23:13:49.431641 4563 ts_tablet_manager.cc:1939] T 930082e074384d88b1df9a98abe4326b P 58dbc937fd8e4d19866a7178b6362fde: Deleting consensus metadata
I20260923 23:13:49.432040 4060 catalog_manager.cc:4841] TS 58dbc937fd8e4d19866a7178b6362fde (127.16.122.131:39105): tablet 930082e074384d88b1df9a98abe4326b (table test-table [id=68582f2d144c4a49a974b75e3b87a118]) successfully deleted
W20260923 23:13:49.434469 4062 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.122.129:43357 (error 108)
I20260923 23:13:49.434935 4537 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:49.434820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.129:43357 (local address 127.0.0.1:55398)
0923 23:13:49.434867 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 23:13:49.434904 (+ 37us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43357: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 23:13:49.434981 4564 ts_tablet_manager.cc:1905] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260923 23:13:49.435070 4062 catalog_manager.cc:4584] TS 39c2969141404a0aa7b179655cafe315 (127.16.122.129:43357): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 930082e074384d88b1df9a98abe4326b: Network error: Client connection negotiation failed: client connection to 127.16.122.129:43357: connect: Connection refused (error 111)
I20260923 23:13:49.436501 4564 ts_tablet_manager.cc:1918] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260923 23:13:49.436574 4564 log.cc:1196] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Deleting WAL directory at /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1790204999454807-16874-0/minicluster-data/ts-1/wal/wals/930082e074384d88b1df9a98abe4326b
I20260923 23:13:49.436836 4564 ts_tablet_manager.cc:1939] T 930082e074384d88b1df9a98abe4326b P 38bb5bc5e9cf42ff8b9c00ae3b080d0b: Deleting consensus metadata
I20260923 23:13:49.437189 4060 catalog_manager.cc:4841] TS 38bb5bc5e9cf42ff8b9c00ae3b080d0b (127.16.122.130:38765): tablet 930082e074384d88b1df9a98abe4326b (table test-table [id=68582f2d144c4a49a974b75e3b87a118]) successfully deleted
I20260923 23:13:49.437417 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4266
I20260923 23:13:49.441449 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4397
I20260923 23:13:49.445092 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4044
2026-09-23T23:13:49Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (729 ms)
[----------] 6 tests from EncryptionPolicyTest (7814 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:49 dist-test-slave-95hh krb5kdc[4568](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:49 dist-test-slave-95hh krb5kdc[4568](info): set up 2 sockets
Sep 23 23:13:49 dist-test-slave-95hh krb5kdc[4568](info): commencing operation
krb5kdc: starting...
W20260923 23:13:50.382625 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:50Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:50.494278 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44443
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45023
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44443
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:50.573457 4581 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:50.573689 4581 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:50.573722 4581 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:50.574887 4581 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:50.574929 4581 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:50.574947 4581 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:50.574962 4581 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:50.574977 4581 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:50.576467 4581 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45023
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44443
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44443
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4581
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:50.576715 4581 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:50.576926 4581 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:50.581426 4586 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:50.581427 4589 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:50.581476 4581 server_base.cc:1034] running on GCE node
W20260923 23:13:50.581455 4587 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:50.581866 4581 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:50.582135 4581 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:50.583271 4581 hybrid_clock.cc:648] HybridClock initialized: now 1790205230583257 us; error 27 us; skew 500 ppm
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4568](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205230, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:50.585625 4581 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:50.586026 4581 webserver.cc:460] Webserver started at http://127.0.0.1:40563/ using document root <none> and password file <none>
I20260923 23:13:50.586216 4581 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:50.586259 4581 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:50.586355 4581 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:50.587478 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "ccfe9dcd71a448fd893ecb0f842a0f46"
format_stamp: "Formatted at 2026-09-23 23:13:50 on dist-test-slave-95hh"
server_key: "2e3ccb428983dd8622e56423fc5eb93a"
server_key_iv: "deb6cf3759977d4589a4f1255ee80a58"
server_key_version: "encryptionkey@0"
I20260923 23:13:50.587788 4581 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "ccfe9dcd71a448fd893ecb0f842a0f46"
format_stamp: "Formatted at 2026-09-23 23:13:50 on dist-test-slave-95hh"
server_key: "2e3ccb428983dd8622e56423fc5eb93a"
server_key_iv: "deb6cf3759977d4589a4f1255ee80a58"
server_key_version: "encryptionkey@0"
I20260923 23:13:50.589077 4581 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:50.589733 4596 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:50.589879 4581 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:50.589957 4581 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "ccfe9dcd71a448fd893ecb0f842a0f46"
format_stamp: "Formatted at 2026-09-23 23:13:50 on dist-test-slave-95hh"
server_key: "2e3ccb428983dd8622e56423fc5eb93a"
server_key_iv: "deb6cf3759977d4589a4f1255ee80a58"
server_key_version: "encryptionkey@0"
I20260923 23:13:50.590014 4581 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:50.605712 4581 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:50.607673 4581 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:50.607789 4581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:50.611449 4581 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44443
I20260923 23:13:50.611482 4648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44443 every 8 connection(s)
I20260923 23:13:50.611824 4581 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:50.612668 4649 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:50.614833 4649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46: Bootstrap starting.
I20260923 23:13:50.615350 4649 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:50.615571 4649 log.cc:824] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:50.616174 4649 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46: No bootstrap required, opened a new log
I20260923 23:13:50.617679 4649 raft_consensus.cc:348] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } }
I20260923 23:13:50.617815 4649 raft_consensus.cc:374] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:50.617836 4649 raft_consensus.cc:729] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccfe9dcd71a448fd893ecb0f842a0f46, State: Initialized, Role: FOLLOWER
I20260923 23:13:50.617950 4649 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } }
I20260923 23:13:50.618023 4649 raft_consensus.cc:388] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:50.618057 4649 raft_consensus.cc:482] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:50.618091 4649 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:50.618547 4649 raft_consensus.cc:504] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } }
I20260923 23:13:50.618654 4649 leader_election.cc:302] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ccfe9dcd71a448fd893ecb0f842a0f46; no voters:
I20260923 23:13:50.618757 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4581
I20260923 23:13:50.618824 4649 leader_election.cc:288] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:50.618861 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:50.618906 4652 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:50.619071 4652 raft_consensus.cc:686] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [term 1 LEADER]: Becoming Leader. State: Replica: ccfe9dcd71a448fd893ecb0f842a0f46, State: Running, Role: LEADER
I20260923 23:13:50.619089 4649 sys_catalog.cc:564] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:50.619112 16874 external_mini_cluster.cc:1428] Setting key 0416e168a3a9f7ac08cf4e09d6749310
I20260923 23:13:50.619148 4652 consensus_queue.cc:237] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } }
I20260923 23:13:50.619609 4654 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ccfe9dcd71a448fd893ecb0f842a0f46. Latest consensus state: current_term: 1 leader_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } } }
I20260923 23:13:50.619675 4653 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ccfe9dcd71a448fd893ecb0f842a0f46" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44443 } } }
I20260923 23:13:50.619777 4654 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:50.619827 4653 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:50.620118 4663 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:50.620848 4663 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:50.623438 4663 catalog_manager.cc:1347] Generated new cluster ID: 477e19d3af504c83b22c1233edc2d04d
I20260923 23:13:50.623487 4663 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4568](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:13:50.630276 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:50.619582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48575 (local address 127.0.0.1:44443)
0923 23:13:50.619747 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:50.619756 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:50.619927 (+ 171us) server_negotiation.cc:407] Connection header received
0923 23:13:50.620066 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:50.620074 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:50.620130 (+ 56us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:50.620259 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:50.621102 (+ 843us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:50.622442 (+ 1340us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:50.623228 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:50.623867 (+ 639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:50.623987 (+ 120us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:50.626684 (+ 2697us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:50.626688 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:50.626688 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:50.626700 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:50.628627 (+ 1927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:50.629052 (+ 425us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:50.629055 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:50.629055 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:50.629173 (+ 118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:50.629389 (+ 216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:50.629390 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:50.629390 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:50.629696 (+ 306us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:50.629831 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:50.629962 (+ 131us) server_negotiation.cc:299] Negotiation successful
0923 23:13:50.630054 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119,"thread_start_us":69,"threads_started":1}
I20260923 23:13:50.631225 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:50.637080 4663 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:50.637655 4663 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:50.639654 4660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:50.632765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57029 (local address 127.0.0.1:44443)
0923 23:13:50.632798 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:50.632800 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:50.632807 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:13:50.632826 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:50.632827 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:50.632836 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:50.632890 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:50.633434 (+ 544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:50.634205 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:50.635177 (+ 972us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:50.635496 (+ 319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:50.635570 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:50.636878 (+ 1308us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:50.636879 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:50.636880 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:50.636887 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:50.638433 (+ 1546us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:50.638865 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:50.638866 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:50.638866 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:50.638931 (+ 65us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:50.639151 (+ 220us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:50.639152 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:50.639153 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:50.639284 (+ 131us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:50.639396 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:50.639490 (+ 94us) server_negotiation.cc:299] Negotiation successful
0923 23:13:50.639562 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:13:50.644129 4663 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ccfe9dcd71a448fd893ecb0f842a0f46: Generated new TSK 0
I20260923 23:13:50.644336 4663 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:50.649860 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4581
W20260923 23:13:50.653631 4566 connection.cc:535] client connection to 127.0.0.1:44443 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44443): Connection reset by peer (error 104)
2026-09-23T23:13:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (1203 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4680](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4680](info): set up 2 sockets
Sep 23 23:13:50 dist-test-slave-95hh krb5kdc[4680](info): commencing operation
krb5kdc: starting...
W20260923 23:13:51.598800 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:51 dist-test-slave-95hh krb5kdc[4680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205231, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:51Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:51.707108 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42273
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45579
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:42273
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:51.787460 4693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:51.787670 4693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:51.787693 4693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:51.788859 4693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:51.788903 4693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:51.788916 4693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:51.788928 4693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:51.788939 4693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:51.790441 4693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45579
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:42273
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:42273
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4693
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:51.790674 4693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:51.790896 4693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:51.795482 4699 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:51.795550 4693 server_base.cc:1034] running on GCE node
W20260923 23:13:51.795496 4698 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:51.795535 4701 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:51.795917 4693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:51.796195 4693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:51.797328 4693 hybrid_clock.cc:648] HybridClock initialized: now 1790205231797313 us; error 30 us; skew 500 ppm
Sep 23 23:13:51 dist-test-slave-95hh krb5kdc[4680](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:51.799630 4693 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:51.800068 4693 webserver.cc:460] Webserver started at http://127.0.0.1:33157/ using document root <none> and password file <none>
I20260923 23:13:51.800279 4693 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:51.800328 4693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:51.800451 4693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:51.801465 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "72553b67a3be43fa95671e82a097a076"
format_stamp: "Formatted at 2026-09-23 23:13:51 on dist-test-slave-95hh"
server_key: "a33790e929fa297a163b0adeafe38541"
server_key_iv: "5680b63218032fcfc77df7d0c9b84bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:51.801788 4693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "72553b67a3be43fa95671e82a097a076"
format_stamp: "Formatted at 2026-09-23 23:13:51 on dist-test-slave-95hh"
server_key: "a33790e929fa297a163b0adeafe38541"
server_key_iv: "5680b63218032fcfc77df7d0c9b84bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:51.802947 4693 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:51.803603 4708 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:51.803743 4693 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:51.803822 4693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "72553b67a3be43fa95671e82a097a076"
format_stamp: "Formatted at 2026-09-23 23:13:51 on dist-test-slave-95hh"
server_key: "a33790e929fa297a163b0adeafe38541"
server_key_iv: "5680b63218032fcfc77df7d0c9b84bf2"
server_key_version: "encryptionkey@0"
I20260923 23:13:51.803889 4693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:51.822959 4693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:51.824826 4693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:51.824975 4693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:51.828694 4693 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:42273
I20260923 23:13:51.828737 4760 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:42273 every 8 connection(s)
I20260923 23:13:51.829073 4693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:51.829893 4761 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:51.831398 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4693
I20260923 23:13:51.831501 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:51.831701 16874 external_mini_cluster.cc:1428] Setting key 891dbac303d003503c1120f485c9af6b
I20260923 23:13:51.832526 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: Bootstrap starting.
I20260923 23:13:51.833201 4761 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:51.833519 4761 log.cc:824] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:51.834264 4761 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: No bootstrap required, opened a new log
I20260923 23:13:51.836397 4761 raft_consensus.cc:348] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } }
I20260923 23:13:51.836581 4761 raft_consensus.cc:374] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:51.836658 4761 raft_consensus.cc:729] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 72553b67a3be43fa95671e82a097a076, State: Initialized, Role: FOLLOWER
I20260923 23:13:51.836799 4761 consensus_queue.cc:260] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } }
I20260923 23:13:51.836892 4761 raft_consensus.cc:388] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:51.836932 4761 raft_consensus.cc:482] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:51.836980 4761 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:51.837550 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:51.832229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49601 (local address 127.0.0.1:42273)
0923 23:13:51.832417 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:51.832427 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:51.832459 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:13:51.832994 (+ 535us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:51.833003 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:51.833063 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:51.833219 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:51.833414 (+ 195us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:51.834701 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:51.835478 (+ 777us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:51.836126 (+ 648us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:51.836243 (+ 117us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:51.836895 (+ 652us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:51.836899 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:51.836900 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:51.837111 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:51.837150 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:51.837165 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:51.837241 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:51.837324 (+ 83us) server_negotiation.cc:299] Negotiation successful
0923 23:13:51.837351 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135,"thread_start_us":87,"threads_started":1}
I20260923 23:13:51.837683 4761 raft_consensus.cc:504] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } }
I20260923 23:13:51.837777 4761 leader_election.cc:302] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 72553b67a3be43fa95671e82a097a076; no voters:
I20260923 23:13:51.837927 4761 leader_election.cc:288] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:51.837970 4766 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:51.838105 4766 raft_consensus.cc:686] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [term 1 LEADER]: Becoming Leader. State: Replica: 72553b67a3be43fa95671e82a097a076, State: Running, Role: LEADER
I20260923 23:13:51.838161 4761 sys_catalog.cc:564] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:51.838275 4766 consensus_queue.cc:237] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } }
I20260923 23:13:51.838632 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:51.838675 4767 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "72553b67a3be43fa95671e82a097a076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } } }
I20260923 23:13:51.838734 4767 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:51.838869 4768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 72553b67a3be43fa95671e82a097a076. Latest consensus state: current_term: 1 leader_uuid: "72553b67a3be43fa95671e82a097a076" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "72553b67a3be43fa95671e82a097a076" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 42273 } } }
I20260923 23:13:51.838908 4768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076 [sys.catalog]: This master's current role is: LEADER
W20260923 23:13:51.839977 4782 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:13:51.840019 4782 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:13:51.840095 4773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:51.840278 4773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:51.841718 4773 catalog_manager.cc:1347] Generated new cluster ID: 9dc38a67d03a419a918419552a7bef9b
I20260923 23:13:51.841759 4773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:51.843644 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:51.840036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48177 (local address 127.0.0.1:42273)
0923 23:13:51.840067 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:51.840070 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:51.840308 (+ 238us) server_negotiation.cc:407] Connection header received
0923 23:13:51.840402 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:51.840403 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:51.840415 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:51.840472 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:51.840602 (+ 130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:51.841331 (+ 729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:51.842381 (+ 1050us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:51.842691 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:51.842763 (+ 72us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:51.843409 (+ 646us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:51.843411 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:51.843411 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:51.843417 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:51.843433 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:51.843437 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:51.843501 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:51.843561 (+ 60us) server_negotiation.cc:299] Negotiation successful
0923 23:13:51.843572 (+ 11us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260923 23:13:51.860828 4773 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:51.861404 4773 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:51.864082 4773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 72553b67a3be43fa95671e82a097a076: Generated new TSK 0
I20260923 23:13:51.864269 4773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:51.869550 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4693
W20260923 23:13:51.873565 4678 connection.cc:535] client connection to 127.0.0.1:42273 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:42273): Connection reset by peer (error 104)
2026-09-23T23:13:51Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (1219 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:51 dist-test-slave-95hh krb5kdc[4792](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:51 dist-test-slave-95hh krb5kdc[4792](info): set up 2 sockets
Sep 23 23:13:51 dist-test-slave-95hh krb5kdc[4792](info): commencing operation
krb5kdc: starting...
W20260923 23:13:52.806756 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.920s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:52 dist-test-slave-95hh krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:52.917716 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:34725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35931
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:52.996042 4805 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:52.996261 4805 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:52.996292 4805 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:52.997426 4805 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:52.997469 4805 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:52.997488 4805 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:52.997503 4805 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:52.997519 4805 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:52.999008 4805 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35931
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4805
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:52.999289 4805 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:52.999490 4805 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:53.004086 4813 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:53.004091 4810 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:53.004088 4811 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:53.004406 4805 server_base.cc:1034] running on GCE node
I20260923 23:13:53.004626 4805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:53.004885 4805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:53.006031 4805 hybrid_clock.cc:648] HybridClock initialized: now 1790205233006019 us; error 30 us; skew 500 ppm
Sep 23 23:13:53 dist-test-slave-95hh krb5kdc[4792](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205233, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:53.008483 4805 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:53.008916 4805 webserver.cc:460] Webserver started at http://127.0.0.1:42085/ using document root <none> and password file <none>
I20260923 23:13:53.009116 4805 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:53.009161 4805 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:53.009291 4805 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:53.010321 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "67c93f926653439187489a6b7e1b1413"
format_stamp: "Formatted at 2026-09-23 23:13:53 on dist-test-slave-95hh"
server_key: "77ae981fc62c191f0a154d0fbb2aeedf"
server_key_iv: "1473c85bf372861329535cf0e0bc8ba2"
server_key_version: "encryptionkey@0"
I20260923 23:13:53.010640 4805 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "67c93f926653439187489a6b7e1b1413"
format_stamp: "Formatted at 2026-09-23 23:13:53 on dist-test-slave-95hh"
server_key: "77ae981fc62c191f0a154d0fbb2aeedf"
server_key_iv: "1473c85bf372861329535cf0e0bc8ba2"
server_key_version: "encryptionkey@0"
I20260923 23:13:53.011927 4805 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:13:53.012573 4820 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:53.012717 4805 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:53.012796 4805 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "67c93f926653439187489a6b7e1b1413"
format_stamp: "Formatted at 2026-09-23 23:13:53 on dist-test-slave-95hh"
server_key: "77ae981fc62c191f0a154d0fbb2aeedf"
server_key_iv: "1473c85bf372861329535cf0e0bc8ba2"
server_key_version: "encryptionkey@0"
I20260923 23:13:53.012856 4805 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:53.016039 4805 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:53.017705 4805 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:53.017829 4805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:53.021653 4805 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35931
I20260923 23:13:53.021692 4872 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35931 every 8 connection(s)
I20260923 23:13:53.022006 4805 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:53.022768 4873 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:53.024979 4873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413: Bootstrap starting.
I20260923 23:13:53.025488 4873 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:53.025730 4873 log.cc:824] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:53.026310 4873 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413: No bootstrap required, opened a new log
I20260923 23:13:53.027873 4873 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260923 23:13:53.028005 4873 raft_consensus.cc:374] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:53.028035 4873 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67c93f926653439187489a6b7e1b1413, State: Initialized, Role: FOLLOWER
I20260923 23:13:53.028160 4873 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260923 23:13:53.028218 4873 raft_consensus.cc:388] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:53.028246 4873 raft_consensus.cc:482] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:53.028291 4873 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:53.028753 4873 raft_consensus.cc:504] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260923 23:13:53.028849 4873 leader_election.cc:302] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 67c93f926653439187489a6b7e1b1413; no voters:
I20260923 23:13:53.028990 4873 leader_election.cc:288] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:53.029013 4876 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:53.029138 4876 raft_consensus.cc:686] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [term 1 LEADER]: Becoming Leader. State: Replica: 67c93f926653439187489a6b7e1b1413, State: Running, Role: LEADER
I20260923 23:13:53.029222 4873 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:53.029260 4876 consensus_queue.cc:237] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } }
I20260923 23:13:53.029603 4878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 67c93f926653439187489a6b7e1b1413. Latest consensus state: current_term: 1 leader_uuid: "67c93f926653439187489a6b7e1b1413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260923 23:13:53.029721 4877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "67c93f926653439187489a6b7e1b1413" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "67c93f926653439187489a6b7e1b1413" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35931 } } }
I20260923 23:13:53.029747 4878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:53.029834 4877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:53.030148 4885 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:53.030638 4885 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:53.031879 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4805
I20260923 23:13:53.031976 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:53.032158 16874 external_mini_cluster.cc:1428] Setting key 5d84b235ec063335203f67259100c4f5
I20260923 23:13:53.032213 4885 catalog_manager.cc:1347] Generated new cluster ID: 1c375aadc5f5437aa734c6c7afa7efc7
I20260923 23:13:53.032256 4885 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:53.034188 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:53.032593 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46973 (local address 127.0.0.1:35931)
0923 23:13:53.032849 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:53.032857 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:53.032879 (+ 22us) server_negotiation.cc:407] Connection header received
0923 23:13:53.032964 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:53.032969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:53.032992 (+ 23us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:53.033096 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:53.033766 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:53.033769 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:53.033769 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:53.033773 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:53.033810 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:53.033820 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:53.033871 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:53.033935 (+ 64us) server_negotiation.cc:299] Negotiation successful
0923 23:13:53.034052 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":217,"thread_start_us":71,"threads_started":1}
I20260923 23:13:53.034948 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:53.037079 4896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:53.036179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34711 (local address 127.0.0.1:35931)
0923 23:13:53.036211 (+ 32us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:53.036212 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:53.036316 (+ 104us) server_negotiation.cc:407] Connection header received
0923 23:13:53.036368 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:53.036368 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:53.036374 (+ 6us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:53.036417 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:53.036896 (+ 479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:53.036897 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:53.036898 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:53.036901 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:53.036913 (+ 12us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:53.036918 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:53.036962 (+ 44us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:53.037010 (+ 48us) server_negotiation.cc:299] Negotiation successful
0923 23:13:53.037023 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:13:53.052201 4885 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:53.052388 4885 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:53.055815 4885 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 67c93f926653439187489a6b7e1b1413: Generated new TSK 0
I20260923 23:13:53.056002 4885 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:53.060335 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4805
2026-09-23T23:13:53Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (1190 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:53 dist-test-slave-95hh krb5kdc[4904](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:53 dist-test-slave-95hh krb5kdc[4904](info): set up 2 sockets
Sep 23 23:13:53 dist-test-slave-95hh krb5kdc[4904](info): commencing operation
krb5kdc: starting...
W20260923 23:13:53.998723 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:54.109841 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38533
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41683
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:54.188717 4917 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:54.188941 4917 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:54.188973 4917 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:54.190111 4917 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:54.190153 4917 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:54.190171 4917 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:54.190186 4917 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:54.190202 4917 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:54.191748 4917 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41683
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38533
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.4917
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:54.191994 4917 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:54.192198 4917 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:54.197134 4923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:54.197151 4922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:54.197403 4925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:54.197906 4917 server_base.cc:1034] running on GCE node
I20260923 23:13:54.198220 4917 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:54.198562 4917 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:54.205885 4917 hybrid_clock.cc:648] HybridClock initialized: now 1790205234205868 us; error 31 us; skew 500 ppm
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[4904](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205234, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:54.211912 4917 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:54.212350 4917 webserver.cc:460] Webserver started at http://127.0.0.1:46377/ using document root <none> and password file <none>
I20260923 23:13:54.212566 4917 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:54.212615 4917 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:54.212741 4917 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:54.213819 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "0eb94c7883844242bf724bf01db1883b"
format_stamp: "Formatted at 2026-09-23 23:13:54 on dist-test-slave-95hh"
server_key: "00b2dd1a4c66c83d688bf6b64d4ece85"
server_key_iv: "35029ef3f84de9f97cab12132a0fa197"
server_key_version: "encryptionkey@0"
I20260923 23:13:54.214155 4917 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "0eb94c7883844242bf724bf01db1883b"
format_stamp: "Formatted at 2026-09-23 23:13:54 on dist-test-slave-95hh"
server_key: "00b2dd1a4c66c83d688bf6b64d4ece85"
server_key_iv: "35029ef3f84de9f97cab12132a0fa197"
server_key_version: "encryptionkey@0"
I20260923 23:13:54.215389 4917 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:13:54.216104 4932 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:54.216259 4917 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:13:54.216347 4917 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "0eb94c7883844242bf724bf01db1883b"
format_stamp: "Formatted at 2026-09-23 23:13:54 on dist-test-slave-95hh"
server_key: "00b2dd1a4c66c83d688bf6b64d4ece85"
server_key_iv: "35029ef3f84de9f97cab12132a0fa197"
server_key_version: "encryptionkey@0"
I20260923 23:13:54.216406 4917 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:54.239890 4917 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:54.242084 4917 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:54.242221 4917 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:54.245919 4917 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38533
I20260923 23:13:54.245966 4984 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38533 every 8 connection(s)
I20260923 23:13:54.246295 4917 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:54.247108 4985 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:54.249197 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b: Bootstrap starting.
I20260923 23:13:54.249692 4985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:54.249905 4985 log.cc:824] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:54.250548 4985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b: No bootstrap required, opened a new log
I20260923 23:13:54.252110 4985 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260923 23:13:54.252243 4985 raft_consensus.cc:374] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:54.252264 4985 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0eb94c7883844242bf724bf01db1883b, State: Initialized, Role: FOLLOWER
I20260923 23:13:54.252368 4985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260923 23:13:54.252440 4985 raft_consensus.cc:388] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:54.252465 4985 raft_consensus.cc:482] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:54.252491 4985 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:54.252941 4985 raft_consensus.cc:504] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260923 23:13:54.253021 4985 leader_election.cc:302] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0eb94c7883844242bf724bf01db1883b; no voters:
I20260923 23:13:54.253150 4985 leader_election.cc:288] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:54.253214 4988 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:54.253356 4985 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:54.253391 4988 raft_consensus.cc:686] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [term 1 LEADER]: Becoming Leader. State: Replica: 0eb94c7883844242bf724bf01db1883b, State: Running, Role: LEADER
I20260923 23:13:54.253482 4988 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } }
I20260923 23:13:54.253886 4989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0eb94c7883844242bf724bf01db1883b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } } }
I20260923 23:13:54.253988 4989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:54.254001 4990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0eb94c7883844242bf724bf01db1883b. Latest consensus state: current_term: 1 leader_uuid: "0eb94c7883844242bf724bf01db1883b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0eb94c7883844242bf724bf01db1883b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38533 } } }
I20260923 23:13:54.254079 4990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:54.254333 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 4917
I20260923 23:13:54.254426 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:54.254446 4997 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:54.254575 16874 external_mini_cluster.cc:1428] Setting key 2a98f730664ce21742a1dc9c6764e4af
I20260923 23:13:54.254896 4997 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:54.256716 4997 catalog_manager.cc:1347] Generated new cluster ID: ff47bf548b4749ebb0987bda91c3a331
I20260923 23:13:54.256767 4997 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[4904](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:13:54.277002 4997 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:54.277489 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:54.255240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44829 (local address 127.0.0.1:38533)
0923 23:13:54.255477 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:54.255485 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:54.255553 (+ 68us) server_negotiation.cc:407] Connection header received
0923 23:13:54.255746 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:54.255757 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:54.255825 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:54.255957 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:54.267229 (+ 11272us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:54.268719 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:54.269881 (+ 1162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:54.270519 (+ 638us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:54.270629 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:54.273586 (+ 2957us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:54.273591 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:54.273592 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:54.273606 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:54.275621 (+ 2015us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:54.276037 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:54.276041 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:54.276041 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:54.276175 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:54.276386 (+ 211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:54.276387 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:54.276387 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:54.276680 (+ 293us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:54.276832 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:54.277109 (+ 277us) server_negotiation.cc:299] Negotiation successful
0923 23:13:54.277260 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":189,"thread_start_us":74,"threads_started":1}
I20260923 23:13:54.277635 4997 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:54.278504 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:54.285741 4997 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 0eb94c7883844242bf724bf01db1883b: Generated new TSK 0
I20260923 23:13:54.285964 4997 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:54.287295 5007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:54.279972 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59391 (local address 127.0.0.1:38533)
0923 23:13:54.280021 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:54.280022 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:54.280155 (+ 133us) server_negotiation.cc:407] Connection header received
0923 23:13:54.280246 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:54.280247 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:54.280258 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:54.280284 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:54.280930 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:54.281707 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:54.282768 (+ 1061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:54.283112 (+ 344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:54.283151 (+ 39us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:54.284619 (+ 1468us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:54.284620 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:54.284621 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:54.284627 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:54.286224 (+ 1597us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:54.286604 (+ 380us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:54.286605 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:54.286605 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:54.286662 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:54.286829 (+ 167us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:54.286829 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:54.286830 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:54.286928 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:54.287003 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:54.287163 (+ 160us) server_negotiation.cc:299] Negotiation successful
0923 23:13:54.287232 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:13:54.289249 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 4917
2026-09-23T23:13:54Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (1229 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[5016](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[5016](info): set up 2 sockets
Sep 23 23:13:54 dist-test-slave-95hh krb5kdc[5016](info): commencing operation
krb5kdc: starting...
W20260923 23:13:55.230824 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:55 dist-test-slave-95hh krb5kdc[5016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205235, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:55.342414 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34967
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33077
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34967
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:55.422206 5029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:55.422430 5029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:55.422463 5029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:55.423691 5029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:55.423735 5029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:55.423755 5029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:55.423770 5029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:55.423787 5029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:55.425315 5029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33077
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34967
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34967
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5029
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:55.425565 5029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:55.425786 5029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:55.430537 5035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:55.430593 5034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:55.430625 5029 server_base.cc:1034] running on GCE node
W20260923 23:13:55.430523 5037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:55.430965 5029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:55.431284 5029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:55.432431 5029 hybrid_clock.cc:648] HybridClock initialized: now 1790205235432416 us; error 32 us; skew 500 ppm
Sep 23 23:13:55 dist-test-slave-95hh krb5kdc[5016](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:55.434772 5029 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:55.435254 5029 webserver.cc:460] Webserver started at http://127.0.0.1:36871/ using document root <none> and password file <none>
I20260923 23:13:55.435474 5029 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:55.435521 5029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:55.435647 5029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:55.436673 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "95ce0abc6115442cb5342e8606c86e1e"
format_stamp: "Formatted at 2026-09-23 23:13:55 on dist-test-slave-95hh"
server_key: "588b873fdcc6656d508f35dd6d76ba22"
server_key_iv: "ec529bd80738aa0cce0158c4438094cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:55.437011 5029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "95ce0abc6115442cb5342e8606c86e1e"
format_stamp: "Formatted at 2026-09-23 23:13:55 on dist-test-slave-95hh"
server_key: "588b873fdcc6656d508f35dd6d76ba22"
server_key_iv: "ec529bd80738aa0cce0158c4438094cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:55.438274 5029 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:55.439049 5044 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:55.439195 5029 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:55.439278 5029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "95ce0abc6115442cb5342e8606c86e1e"
format_stamp: "Formatted at 2026-09-23 23:13:55 on dist-test-slave-95hh"
server_key: "588b873fdcc6656d508f35dd6d76ba22"
server_key_iv: "ec529bd80738aa0cce0158c4438094cb"
server_key_version: "encryptionkey@0"
I20260923 23:13:55.439345 5029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:55.458171 5029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:55.460220 5029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:55.460376 5029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:55.464227 5029 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34967
I20260923 23:13:55.464337 5096 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34967 every 8 connection(s)
I20260923 23:13:55.464623 5029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:55.465477 5097 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:55.467065 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5029
I20260923 23:13:55.467173 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:55.467370 16874 external_mini_cluster.cc:1428] Setting key 72a1ad15f6ec4f477aa51ff7475c9008
I20260923 23:13:55.467943 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e: Bootstrap starting.
I20260923 23:13:55.468520 5097 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:55.468772 5097 log.cc:824] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:55.469388 5097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e: No bootstrap required, opened a new log
I20260923 23:13:55.471122 5097 raft_consensus.cc:348] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260923 23:13:55.471252 5097 raft_consensus.cc:374] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:55.471283 5097 raft_consensus.cc:729] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 95ce0abc6115442cb5342e8606c86e1e, State: Initialized, Role: FOLLOWER
I20260923 23:13:55.471421 5097 consensus_queue.cc:260] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260923 23:13:55.471503 5097 raft_consensus.cc:388] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:55.471540 5097 raft_consensus.cc:482] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:55.471585 5097 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:55.472096 5097 raft_consensus.cc:504] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260923 23:13:55.472190 5097 leader_election.cc:302] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 95ce0abc6115442cb5342e8606c86e1e; no voters:
I20260923 23:13:55.472342 5097 leader_election.cc:288] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:55.472362 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:55.467900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58975 (local address 127.0.0.1:34967)
0923 23:13:55.468091 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:55.468100 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:55.468130 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:13:55.468556 (+ 426us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:55.468563 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:55.468617 (+ 54us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:55.468754 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:55.468944 (+ 190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:55.469844 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:55.470575 (+ 731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:55.470982 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:55.471109 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:55.471745 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:55.471749 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:55.471749 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:55.471922 (+ 173us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:55.471952 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:55.471962 (+ 10us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:55.472013 (+ 51us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:55.472098 (+ 85us) server_negotiation.cc:299] Negotiation successful
0923 23:13:55.472164 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":143,"thread_start_us":77,"threads_started":1}
I20260923 23:13:55.472384 5102 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:55.472482 5102 raft_consensus.cc:686] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [term 1 LEADER]: Becoming Leader. State: Replica: 95ce0abc6115442cb5342e8606c86e1e, State: Running, Role: LEADER
I20260923 23:13:55.472575 5102 consensus_queue.cc:237] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } }
I20260923 23:13:55.472584 5097 sys_catalog.cc:564] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:55.472955 5103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 95ce0abc6115442cb5342e8606c86e1e. Latest consensus state: current_term: 1 leader_uuid: "95ce0abc6115442cb5342e8606c86e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } } }
I20260923 23:13:55.473026 5103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:55.473201 5107 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:55.473263 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:55.473683 5107 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:55.474212 5104 sys_catalog.cc:455] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "95ce0abc6115442cb5342e8606c86e1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "95ce0abc6115442cb5342e8606c86e1e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34967 } } }
I20260923 23:13:55.474306 5104 sys_catalog.cc:458] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:55.475214 5107 catalog_manager.cc:1347] Generated new cluster ID: 720ed4d6e4814959a44a8b9e1b258531
I20260923 23:13:55.475265 5107 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:55.478317 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:55.474711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48839 (local address 127.0.0.1:34967)
0923 23:13:55.474742 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:55.474744 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:55.475163 (+ 419us) server_negotiation.cc:407] Connection header received
0923 23:13:55.475220 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:55.475220 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:55.475230 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:55.475289 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:55.475466 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:55.476156 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:55.477138 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:55.477416 (+ 278us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:55.477467 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:55.478108 (+ 641us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:55.478109 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:55.478109 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:55.478113 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:55.478127 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:55.478131 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:55.478164 (+ 33us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:55.478227 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 23:13:55.478248 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14}
I20260923 23:13:55.504686 5107 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:55.505281 5107 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:55.510653 5107 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 95ce0abc6115442cb5342e8606c86e1e: Generated new TSK 0
I20260923 23:13:55.510852 5107 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:55.522280 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5029
2026-09-23T23:13:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (1233 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:55 dist-test-slave-95hh krb5kdc[5128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:55 dist-test-slave-95hh krb5kdc[5128](info): set up 2 sockets
Sep 23 23:13:55 dist-test-slave-95hh krb5kdc[5128](info): commencing operation
krb5kdc: starting...
W20260923 23:13:56.472689 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.933s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:56 dist-test-slave-95hh krb5kdc[5128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:56Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:56Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:56.583912 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33277
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:39713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33277
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:56.663784 5141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:56.664000 5141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:56.664021 5141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:56.665174 5141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:56.665218 5141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:56.665230 5141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:56.665241 5141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:56.665252 5141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:56.666644 5141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33277
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33277
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5141
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:56.666842 5141 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:56.667011 5141 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:13:56.671706 5147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:56.671823 5141 server_base.cc:1034] running on GCE node
W20260923 23:13:56.673305 5146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:56.673312 5149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:56.673607 5141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:56.673887 5141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:56.675052 5141 hybrid_clock.cc:648] HybridClock initialized: now 1790205236675016 us; error 35 us; skew 500 ppm
Sep 23 23:13:56 dist-test-slave-95hh krb5kdc[5128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:56.678637 5141 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:56.679222 5141 webserver.cc:460] Webserver started at http://127.0.0.1:39571/ using document root <none> and password file <none>
I20260923 23:13:56.679450 5141 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:56.679497 5141 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:56.679622 5141 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:56.680786 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "a4e816604c3444b7ae090192fb9170d8"
format_stamp: "Formatted at 2026-09-23 23:13:56 on dist-test-slave-95hh"
server_key: "6a025583d77d580d1c1e58ce1c0cf76f"
server_key_iv: "5280b097dbd497a3a1b225a7bd6158ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:56.681121 5141 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "a4e816604c3444b7ae090192fb9170d8"
format_stamp: "Formatted at 2026-09-23 23:13:56 on dist-test-slave-95hh"
server_key: "6a025583d77d580d1c1e58ce1c0cf76f"
server_key_iv: "5280b097dbd497a3a1b225a7bd6158ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:56.682407 5141 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:13:56.683131 5156 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:56.683271 5141 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:56.683352 5141 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "a4e816604c3444b7ae090192fb9170d8"
format_stamp: "Formatted at 2026-09-23 23:13:56 on dist-test-slave-95hh"
server_key: "6a025583d77d580d1c1e58ce1c0cf76f"
server_key_iv: "5280b097dbd497a3a1b225a7bd6158ed"
server_key_version: "encryptionkey@0"
I20260923 23:13:56.683418 5141 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:56.686779 5141 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:56.688937 5141 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:56.689064 5141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:56.692961 5141 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33277
I20260923 23:13:56.692991 5208 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33277 every 8 connection(s)
I20260923 23:13:56.693336 5141 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:56.694176 5209 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:56.696460 5209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8: Bootstrap starting.
I20260923 23:13:56.697028 5209 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:56.697268 5209 log.cc:824] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:56.698046 5209 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8: No bootstrap required, opened a new log
I20260923 23:13:56.698191 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5141
I20260923 23:13:56.698300 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:56.698539 16874 external_mini_cluster.cc:1428] Setting key 40287fa9fd577227363472e43626dd45
I20260923 23:13:56.700109 5209 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260923 23:13:56.700273 5209 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:56.700347 5209 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4e816604c3444b7ae090192fb9170d8, State: Initialized, Role: FOLLOWER
I20260923 23:13:56.700515 5209 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260923 23:13:56.700616 5209 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:56.700644 5209 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:56.700693 5209 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:56.701478 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:56.699186 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53843 (local address 127.0.0.1:33277)
0923 23:13:56.699366 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:56.699373 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:56.699400 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:13:56.700087 (+ 687us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:56.700094 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:56.700122 (+ 28us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:56.700277 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:56.701058 (+ 781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:56.701061 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:56.701061 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:56.701065 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:56.701103 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:56.701114 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:56.701161 (+ 47us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:56.701235 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:13:56.701270 (+ 35us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128,"thread_start_us":85,"threads_started":1}
I20260923 23:13:56.701808 5209 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260923 23:13:56.701915 5209 leader_election.cc:302] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4e816604c3444b7ae090192fb9170d8; no voters:
I20260923 23:13:56.702075 5209 leader_election.cc:288] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:56.702178 5214 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:56.702327 5209 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:56.702337 5214 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [term 1 LEADER]: Becoming Leader. State: Replica: a4e816604c3444b7ae090192fb9170d8, State: Running, Role: LEADER
I20260923 23:13:56.702463 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:56.702534 5214 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } }
I20260923 23:13:56.703112 5215 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4e816604c3444b7ae090192fb9170d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } } }
I20260923 23:13:56.703203 5215 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:56.703169 5216 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4e816604c3444b7ae090192fb9170d8. Latest consensus state: current_term: 1 leader_uuid: "a4e816604c3444b7ae090192fb9170d8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4e816604c3444b7ae090192fb9170d8" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33277 } } }
I20260923 23:13:56.703250 5216 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:56.703917 5223 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:56.704447 5223 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:56.706094 5223 catalog_manager.cc:1347] Generated new cluster ID: a913ff7d3cc545fba4e7bb31e84416e5
I20260923 23:13:56.706143 5223 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:56.706326 5213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:56.704084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55903 (local address 127.0.0.1:33277)
0923 23:13:56.704118 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:56.704119 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:56.705306 (+ 1187us) server_negotiation.cc:407] Connection header received
0923 23:13:56.705386 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:56.705387 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:56.705392 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:56.705445 (+ 53us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:56.706085 (+ 640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:56.706086 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:56.706086 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:56.706090 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:56.706104 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:56.706107 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:56.706152 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:56.706250 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 23:13:56.706263 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:13:56.717965 5223 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:56.718137 5223 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:56.729246 5223 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4e816604c3444b7ae090192fb9170d8: Generated new TSK 0
I20260923 23:13:56.729425 5223 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:56.733136 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5141
2026-09-23T23:13:56Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (1211 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:56 dist-test-slave-95hh krb5kdc[5240](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:56 dist-test-slave-95hh krb5kdc[5240](info): set up 2 sockets
Sep 23 23:13:56 dist-test-slave-95hh krb5kdc[5240](info): commencing operation
krb5kdc: starting...
W20260923 23:13:57.682891 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:57 dist-test-slave-95hh krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:57.794052 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36697
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36697
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:57.892982 5253 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:57.893214 5253 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:57.893245 5253 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:57.894408 5253 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:57.894459 5253 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:57.894476 5253 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:57.894490 5253 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:57.894507 5253 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:57.896062 5253 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36697
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36697
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5253
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:57.896324 5253 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:57.896538 5253 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:57.901597 5253 server_base.cc:1034] running on GCE node
W20260923 23:13:57.901667 5258 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:57.901769 5261 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:57.901682 5259 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:57.902012 5253 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:57.902393 5253 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:57.903541 5253 hybrid_clock.cc:648] HybridClock initialized: now 1790205237903503 us; error 54 us; skew 500 ppm
Sep 23 23:13:57 dist-test-slave-95hh krb5kdc[5240](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205237, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:57.906029 5253 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:57.906533 5253 webserver.cc:460] Webserver started at http://127.0.0.1:43815/ using document root <none> and password file <none>
I20260923 23:13:57.906733 5253 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:57.906774 5253 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:57.906899 5253 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:57.907997 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "fe3f6f3e982e4d07871d157732426f14"
format_stamp: "Formatted at 2026-09-23 23:13:57 on dist-test-slave-95hh"
server_key: "f4d1d5456dc01ab1e216c5f7d0b1a2e7"
server_key_iv: "3cc49de6109186182f8633cc7b03cf08"
server_key_version: "encryptionkey@0"
I20260923 23:13:57.908371 5253 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "fe3f6f3e982e4d07871d157732426f14"
format_stamp: "Formatted at 2026-09-23 23:13:57 on dist-test-slave-95hh"
server_key: "f4d1d5456dc01ab1e216c5f7d0b1a2e7"
server_key_iv: "3cc49de6109186182f8633cc7b03cf08"
server_key_version: "encryptionkey@0"
I20260923 23:13:57.909569 5253 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:13:57.910223 5268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:57.910415 5253 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:13:57.910506 5253 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "fe3f6f3e982e4d07871d157732426f14"
format_stamp: "Formatted at 2026-09-23 23:13:57 on dist-test-slave-95hh"
server_key: "f4d1d5456dc01ab1e216c5f7d0b1a2e7"
server_key_iv: "3cc49de6109186182f8633cc7b03cf08"
server_key_version: "encryptionkey@0"
I20260923 23:13:57.910560 5253 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:57.933184 5253 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:57.936435 5253 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:57.936565 5253 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:57.940346 5253 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36697
I20260923 23:13:57.940385 5320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36697 every 8 connection(s)
I20260923 23:13:57.940719 5253 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:57.941560 5321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:57.944499 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14: Bootstrap starting.
I20260923 23:13:57.945082 5321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:57.945331 5321 log.cc:824] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:57.945986 5321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14: No bootstrap required, opened a new log
I20260923 23:13:57.947590 5321 raft_consensus.cc:348] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260923 23:13:57.947746 5321 raft_consensus.cc:374] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:57.947770 5321 raft_consensus.cc:729] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fe3f6f3e982e4d07871d157732426f14, State: Initialized, Role: FOLLOWER
I20260923 23:13:57.947909 5321 consensus_queue.cc:260] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260923 23:13:57.947983 5321 raft_consensus.cc:388] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:57.948024 5321 raft_consensus.cc:482] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:57.948071 5321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:57.948561 5321 raft_consensus.cc:504] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260923 23:13:57.948662 5321 leader_election.cc:302] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: fe3f6f3e982e4d07871d157732426f14; no voters:
I20260923 23:13:57.948822 5321 leader_election.cc:288] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:57.948856 5324 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:57.948932 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5253
I20260923 23:13:57.949008 5324 raft_consensus.cc:686] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [term 1 LEADER]: Becoming Leader. State: Replica: fe3f6f3e982e4d07871d157732426f14, State: Running, Role: LEADER
I20260923 23:13:57.949044 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:57.949079 5321 sys_catalog.cc:564] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:57.949090 5324 consensus_queue.cc:237] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } }
I20260923 23:13:57.949255 16874 external_mini_cluster.cc:1428] Setting key defbff6f47ea309bc83cefddfa9b88cd
I20260923 23:13:57.949420 5324 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "fe3f6f3e982e4d07871d157732426f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } } }
I20260923 23:13:57.949558 5324 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:57.949717 5324 sys_catalog.cc:455] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fe3f6f3e982e4d07871d157732426f14. Latest consensus state: current_term: 1 leader_uuid: "fe3f6f3e982e4d07871d157732426f14" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "fe3f6f3e982e4d07871d157732426f14" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36697 } } }
I20260923 23:13:57.949819 5324 sys_catalog.cc:458] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:57.950245 5332 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:57.950785 5332 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:57.955453 5332 catalog_manager.cc:1347] Generated new cluster ID: 3f4e57e90b3142b19c62e3d6b3234bac
I20260923 23:13:57.955502 5332 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:13:57 dist-test-slave-95hh krb5kdc[5240](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205237, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:13:57.961921 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:57.949985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:51185 (local address 127.0.0.1:36697)
0923 23:13:57.950158 (+ 173us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:57.950168 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:57.950198 (+ 30us) server_negotiation.cc:407] Connection header received
0923 23:13:57.950317 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:57.950324 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:57.950384 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:57.950529 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:57.951287 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:57.952449 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:57.953490 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:57.953888 (+ 398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:57.953959 (+ 71us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:57.956888 (+ 2929us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:57.956893 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:57.956893 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:57.956906 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:57.959797 (+ 2891us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:57.960243 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:57.960248 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:57.960248 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:57.960375 (+ 127us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:57.960743 (+ 368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:57.960744 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:57.960744 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:57.961280 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:57.961458 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:57.961550 (+ 92us) server_negotiation.cc:299] Negotiation successful
0923 23:13:57.961660 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123,"thread_start_us":73,"threads_started":1}
I20260923 23:13:57.963234 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:57.974474 5336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:57.964817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:53887 (local address 127.0.0.1:36697)
0923 23:13:57.964856 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:57.964858 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:57.967244 (+ 2386us) server_negotiation.cc:407] Connection header received
0923 23:13:57.967279 (+ 35us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:57.967280 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:57.967293 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:57.967351 (+ 58us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:57.967967 (+ 616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:57.968755 (+ 788us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:57.969713 (+ 958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:57.970171 (+ 458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:57.970279 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:57.971732 (+ 1453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:57.971734 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:57.971735 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:13:57.971743 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:57.973280 (+ 1537us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:57.973683 (+ 403us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:57.973684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:57.973684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:57.973750 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:13:57.973941 (+ 191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:13:57.973942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:13:57.973942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:13:57.974090 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:57.974203 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:57.974308 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 23:13:57.974371 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:13:57.991531 5332 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:57.992061 5332 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:57.997725 5332 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fe3f6f3e982e4d07871d157732426f14: Generated new TSK 0
I20260923 23:13:57.997910 5332 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:58.004424 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5253
W20260923 23:13:58.008242 5238 connection.cc:535] client connection to 127.0.0.1:36697 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36697): Connection reset by peer (error 104)
2026-09-23T23:13:58Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (1270 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:58 dist-test-slave-95hh krb5kdc[5352](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:58 dist-test-slave-95hh krb5kdc[5352](info): set up 2 sockets
Sep 23 23:13:58 dist-test-slave-95hh krb5kdc[5352](info): commencing operation
krb5kdc: starting...
W20260923 23:13:58.946841 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:13:59 dist-test-slave-95hh krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:13:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:13:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:13:59.057503 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34859
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33967
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34859
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:13:59.138263 5365 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:13:59.138494 5365 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:13:59.138525 5365 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:13:59.139775 5365 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:13:59.139823 5365 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:13:59.139844 5365 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:13:59.139860 5365 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:13:59.139875 5365 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:13:59.141396 5365 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33967
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34859
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34859
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5365
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:13:59.141644 5365 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:13:59.141860 5365 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:13:59.147166 5365 server_base.cc:1034] running on GCE node
W20260923 23:13:59.147123 5373 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:59.147128 5370 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:13:59.147106 5371 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:13:59.147514 5365 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:13:59.147821 5365 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:13:59.148957 5365 hybrid_clock.cc:648] HybridClock initialized: now 1790205239148945 us; error 29 us; skew 500 ppm
Sep 23 23:13:59 dist-test-slave-95hh krb5kdc[5352](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:13:59.151301 5365 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:13:59.151814 5365 webserver.cc:460] Webserver started at http://127.0.0.1:43217/ using document root <none> and password file <none>
I20260923 23:13:59.152047 5365 fs_manager.cc:359] Metadata directory not provided
I20260923 23:13:59.152096 5365 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:13:59.152221 5365 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:13:59.153275 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6"
format_stamp: "Formatted at 2026-09-23 23:13:59 on dist-test-slave-95hh"
server_key: "70f5411706d49df7139286a72bd8d269"
server_key_iv: "ae11223137863c57158ed5ad3fbe0cb3"
server_key_version: "encryptionkey@0"
I20260923 23:13:59.153615 5365 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6"
format_stamp: "Formatted at 2026-09-23 23:13:59 on dist-test-slave-95hh"
server_key: "70f5411706d49df7139286a72bd8d269"
server_key_iv: "ae11223137863c57158ed5ad3fbe0cb3"
server_key_version: "encryptionkey@0"
I20260923 23:13:59.154874 5365 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:13:59.155766 5380 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:13:59.155942 5365 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:13:59.156046 5365 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6"
format_stamp: "Formatted at 2026-09-23 23:13:59 on dist-test-slave-95hh"
server_key: "70f5411706d49df7139286a72bd8d269"
server_key_iv: "ae11223137863c57158ed5ad3fbe0cb3"
server_key_version: "encryptionkey@0"
I20260923 23:13:59.156118 5365 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:13:59.165323 5365 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:13:59.167004 5365 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:13:59.167158 5365 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:13:59.170814 5365 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34859
I20260923 23:13:59.170846 5432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34859 every 8 connection(s)
I20260923 23:13:59.171243 5365 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:13:59.171936 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5365
I20260923 23:13:59.172044 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:13:59.172142 5433 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:13:59.172271 16874 external_mini_cluster.cc:1428] Setting key 5adf6b3d2cfeb7dd39b8ac8d01f2f843
I20260923 23:13:59.174970 5433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6: Bootstrap starting.
I20260923 23:13:59.175678 5433 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6: Neither blocks nor log segments found. Creating new log.
I20260923 23:13:59.175943 5433 log.cc:824] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6: Log is configured to *not* fsync() on all Append() calls
I20260923 23:13:59.176560 5433 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6: No bootstrap required, opened a new log
I20260923 23:13:59.177609 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:59.172848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:47811 (local address 127.0.0.1:34859)
0923 23:13:59.173002 (+ 154us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:59.173011 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:59.173037 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:13:59.173493 (+ 456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:59.173500 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:59.173549 (+ 49us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:59.173662 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:59.173863 (+ 201us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:59.174724 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:59.175758 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:59.176215 (+ 457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:59.176315 (+ 100us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:59.177058 (+ 743us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:59.177062 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:59.177062 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:59.177214 (+ 152us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:59.177246 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:59.177257 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:59.177325 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:59.177416 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 23:13:59.177443 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98,"thread_start_us":53,"threads_started":1}
I20260923 23:13:59.178550 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:13:59.178609 5433 raft_consensus.cc:348] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } }
I20260923 23:13:59.178762 5433 raft_consensus.cc:374] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:13:59.178797 5433 raft_consensus.cc:729] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23fea0b2f4ca4d0486bf0900d6adb8a6, State: Initialized, Role: FOLLOWER
I20260923 23:13:59.178911 5433 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } }
I20260923 23:13:59.178982 5433 raft_consensus.cc:388] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:13:59.179001 5433 raft_consensus.cc:482] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:13:59.179054 5433 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:13:59.179629 5433 raft_consensus.cc:504] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } }
I20260923 23:13:59.179719 5433 leader_election.cc:302] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 23fea0b2f4ca4d0486bf0900d6adb8a6; no voters:
I20260923 23:13:59.179872 5433 leader_election.cc:288] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:13:59.179900 5442 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:13:59.180178 5442 raft_consensus.cc:686] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [term 1 LEADER]: Becoming Leader. State: Replica: 23fea0b2f4ca4d0486bf0900d6adb8a6, State: Running, Role: LEADER
I20260923 23:13:59.180203 5433 sys_catalog.cc:564] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:13:59.180344 5442 consensus_queue.cc:237] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } }
I20260923 23:13:59.180692 5445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 23fea0b2f4ca4d0486bf0900d6adb8a6. Latest consensus state: current_term: 1 leader_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } } }
I20260923 23:13:59.180904 5445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:59.181329 5452 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:13:59.181829 5452 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:13:59.180840 5444 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "23fea0b2f4ca4d0486bf0900d6adb8a6" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34859 } } }
I20260923 23:13:59.182336 5444 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6 [sys.catalog]: This master's current role is: LEADER
I20260923 23:13:59.183449 5452 catalog_manager.cc:1347] Generated new cluster ID: 8b68ce3e4e194f6c9debe2fe0045e3d6
I20260923 23:13:59.183434 5436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:13:59.179975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:40333 (local address 127.0.0.1:34859)
0923 23:13:59.180035 (+ 60us) server_negotiation.cc:206] Beginning negotiation
0923 23:13:59.180037 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:13:59.180113 (+ 76us) server_negotiation.cc:407] Connection header received
0923 23:13:59.180231 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:13:59.180231 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:13:59.180242 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:13:59.180294 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:13:59.180471 (+ 177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:59.181142 (+ 671us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:59.182160 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:13:59.182436 (+ 276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:13:59.182510 (+ 74us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:13:59.183194 (+ 684us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:13:59.183195 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:13:59.183196 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:13:59.183200 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:13:59.183216 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:13:59.183220 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:13:59.183279 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:13:59.183354 (+ 75us) server_negotiation.cc:299] Negotiation successful
0923 23:13:59.183364 (+ 10us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260923 23:13:59.183513 5452 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:13:59.192086 5452 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:13:59.192575 5452 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:13:59.196326 5452 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 23fea0b2f4ca4d0486bf0900d6adb8a6: Generated new TSK 0
I20260923 23:13:59.196480 5452 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:13:59.200724 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5365
W20260923 23:13:59.204774 5350 connection.cc:535] client connection to 127.0.0.1:34859 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34859): Connection reset by peer (error 104)
2026-09-23T23:13:59Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (1196 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:13:59 dist-test-slave-95hh krb5kdc[5464](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:13:59 dist-test-slave-95hh krb5kdc[5464](info): set up 2 sockets
Sep 23 23:13:59 dist-test-slave-95hh krb5kdc[5464](info): commencing operation
krb5kdc: starting...
W20260923 23:14:00.150805 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.932s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:00 dist-test-slave-95hh krb5kdc[5464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:14:00.261359 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39133
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35697
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39133
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:00.339960 5477 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:00.340181 5477 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:00.340212 5477 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:00.341394 5477 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:00.341439 5477 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:00.341457 5477 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:00.341473 5477 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:00.341488 5477 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:00.342988 5477 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35697
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39133
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39133
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5477
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:00.343276 5477 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:00.343513 5477 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:00.348269 5483 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:00.348289 5485 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:00.348285 5482 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:00.348608 5477 server_base.cc:1034] running on GCE node
I20260923 23:14:00.348830 5477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:00.349109 5477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:00.350250 5477 hybrid_clock.cc:648] HybridClock initialized: now 1790205240350240 us; error 34 us; skew 500 ppm
Sep 23 23:14:00 dist-test-slave-95hh krb5kdc[5464](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:00.352735 5477 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:14:00.353269 5477 webserver.cc:460] Webserver started at http://127.0.0.1:43605/ using document root <none> and password file <none>
I20260923 23:14:00.353499 5477 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:00.353549 5477 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:00.353675 5477 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:00.354745 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "6f6c48995de144fab0f698f943fad558"
format_stamp: "Formatted at 2026-09-23 23:14:00 on dist-test-slave-95hh"
server_key: "9766659a5d541459ed7b3a60b6388426"
server_key_iv: "fed8aa329da27d22178a1134d03177e9"
server_key_version: "encryptionkey@0"
I20260923 23:14:00.355122 5477 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "6f6c48995de144fab0f698f943fad558"
format_stamp: "Formatted at 2026-09-23 23:14:00 on dist-test-slave-95hh"
server_key: "9766659a5d541459ed7b3a60b6388426"
server_key_iv: "fed8aa329da27d22178a1134d03177e9"
server_key_version: "encryptionkey@0"
I20260923 23:14:00.356359 5477 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:14:00.356992 5492 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:00.357139 5477 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:00.357218 5477 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "6f6c48995de144fab0f698f943fad558"
format_stamp: "Formatted at 2026-09-23 23:14:00 on dist-test-slave-95hh"
server_key: "9766659a5d541459ed7b3a60b6388426"
server_key_iv: "fed8aa329da27d22178a1134d03177e9"
server_key_version: "encryptionkey@0"
I20260923 23:14:00.357286 5477 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:00.360544 5477 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:00.362677 5477 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:00.362805 5477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:00.366523 5477 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39133
I20260923 23:14:00.366551 5544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39133 every 8 connection(s)
I20260923 23:14:00.366858 5477 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:00.367686 5545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:00.369865 5545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558: Bootstrap starting.
I20260923 23:14:00.370384 5545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:00.370597 5545 log.cc:824] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:00.371245 5545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558: No bootstrap required, opened a new log
I20260923 23:14:00.372793 5545 raft_consensus.cc:348] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260923 23:14:00.372927 5545 raft_consensus.cc:374] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:00.372948 5545 raft_consensus.cc:729] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6f6c48995de144fab0f698f943fad558, State: Initialized, Role: FOLLOWER
I20260923 23:14:00.373056 5545 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260923 23:14:00.373127 5545 raft_consensus.cc:388] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:00.373149 5545 raft_consensus.cc:482] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:00.373181 5545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:00.373644 5545 raft_consensus.cc:504] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260923 23:14:00.373725 5545 leader_election.cc:302] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6f6c48995de144fab0f698f943fad558; no voters:
I20260923 23:14:00.373844 5545 leader_election.cc:288] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:00.373919 5548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:00.374058 5545 sys_catalog.cc:564] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:00.374068 5548 raft_consensus.cc:686] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [term 1 LEADER]: Becoming Leader. State: Replica: 6f6c48995de144fab0f698f943fad558, State: Running, Role: LEADER
I20260923 23:14:00.374172 5548 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } }
I20260923 23:14:00.374538 5550 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6f6c48995de144fab0f698f943fad558. Latest consensus state: current_term: 1 leader_uuid: "6f6c48995de144fab0f698f943fad558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } } }
I20260923 23:14:00.374656 5550 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:00.374562 5549 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6f6c48995de144fab0f698f943fad558" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6f6c48995de144fab0f698f943fad558" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39133 } } }
I20260923 23:14:00.374874 5549 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:00.375047 5552 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:00.375522 5552 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:00.375770 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5477
I20260923 23:14:00.375867 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:00.376006 16874 external_mini_cluster.cc:1428] Setting key bd4c4fb0777e3e73c751104a9c12ae0c
I20260923 23:14:00.377032 5552 catalog_manager.cc:1347] Generated new cluster ID: c723a99ef7b84560875ab2134abb8478
I20260923 23:14:00.377085 5552 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:00.378170 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:00.376542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:54151 (local address 127.0.0.1:39133)
0923 23:14:00.376693 (+ 151us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:00.376703 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:00.376826 (+ 123us) server_negotiation.cc:407] Connection header received
0923 23:14:00.376962 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:00.376969 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:00.377000 (+ 31us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:00.377139 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:00.377778 (+ 639us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:00.377781 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:00.377782 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:00.377785 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:00.377824 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:00.377837 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:00.377893 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:00.377963 (+ 70us) server_negotiation.cc:299] Negotiation successful
0923 23:14:00.378005 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104,"thread_start_us":79,"threads_started":1}
I20260923 23:14:00.379887 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:00.382345 5567 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:00.381330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:32961 (local address 127.0.0.1:39133)
0923 23:14:00.381365 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:00.381367 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:00.381491 (+ 124us) server_negotiation.cc:407] Connection header received
0923 23:14:00.381553 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:00.381554 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:00.381559 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:00.381609 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:00.382161 (+ 552us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:00.382161 (+ 0us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:00.382162 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:00.382165 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:00.382180 (+ 15us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:00.382183 (+ 3us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:00.382224 (+ 41us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:00.382275 (+ 51us) server_negotiation.cc:299] Negotiation successful
0923 23:14:00.382287 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:00.388229 5552 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:00.388388 5552 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:00.394925 5552 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 6f6c48995de144fab0f698f943fad558: Generated new TSK 0
I20260923 23:14:00.395126 5552 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:00.402787 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5477
2026-09-23T23:14:00Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:00 dist-test-slave-95hh krb5kdc[5576](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:00 dist-test-slave-95hh krb5kdc[5576](info): set up 2 sockets
Sep 23 23:14:00 dist-test-slave-95hh krb5kdc[5576](info): commencing operation
krb5kdc: starting...
W20260923 23:14:01.342832 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.923s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:01Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:01Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:14:01.453235 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34809
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45597
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34809
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:01.532039 5589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:01.532251 5589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:01.532284 5589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:01.533437 5589 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:01.533480 5589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:01.533499 5589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:01.533515 5589 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:01.533530 5589 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:01.535001 5589 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45597
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34809
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34809
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5589
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:01.535235 5589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:01.535414 5589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:01.539974 5595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:01.539989 5597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:01.539983 5594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:01.540226 5589 server_base.cc:1034] running on GCE node
I20260923 23:14:01.540454 5589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:01.540745 5589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:01.541898 5589 hybrid_clock.cc:648] HybridClock initialized: now 1790205241541879 us; error 32 us; skew 500 ppm
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5576](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205241, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:01.544344 5589 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:14:01.544906 5589 webserver.cc:460] Webserver started at http://127.0.0.1:44727/ using document root <none> and password file <none>
I20260923 23:14:01.545113 5589 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:01.545161 5589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:01.545288 5589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:01.546346 5589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "9a3caf93379c4e8e9526de61cc1cf495"
format_stamp: "Formatted at 2026-09-23 23:14:01 on dist-test-slave-95hh"
server_key: "d03de691fc20ff1278ac10efe0036995"
server_key_iv: "bb3ecb3bec2d748f0f67f78ea4affee9"
server_key_version: "encryptionkey@0"
I20260923 23:14:01.546675 5589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "9a3caf93379c4e8e9526de61cc1cf495"
format_stamp: "Formatted at 2026-09-23 23:14:01 on dist-test-slave-95hh"
server_key: "d03de691fc20ff1278ac10efe0036995"
server_key_iv: "bb3ecb3bec2d748f0f67f78ea4affee9"
server_key_version: "encryptionkey@0"
I20260923 23:14:01.547960 5589 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:14:01.548637 5604 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:01.548794 5589 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:01.548878 5589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "9a3caf93379c4e8e9526de61cc1cf495"
format_stamp: "Formatted at 2026-09-23 23:14:01 on dist-test-slave-95hh"
server_key: "d03de691fc20ff1278ac10efe0036995"
server_key_iv: "bb3ecb3bec2d748f0f67f78ea4affee9"
server_key_version: "encryptionkey@0"
I20260923 23:14:01.548938 5589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:01.562916 5589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:01.565212 5589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:01.565362 5589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:01.569247 5589 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34809
I20260923 23:14:01.569288 5656 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:34809 every 8 connection(s)
I20260923 23:14:01.569613 5589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:01.570415 5657 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:01.572726 5657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495: Bootstrap starting.
I20260923 23:14:01.573238 5657 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:01.573467 5657 log.cc:824] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:01.574079 5657 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495: No bootstrap required, opened a new log
I20260923 23:14:01.575630 5657 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260923 23:14:01.575796 5657 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:01.575827 5657 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9a3caf93379c4e8e9526de61cc1cf495, State: Initialized, Role: FOLLOWER
I20260923 23:14:01.575927 5657 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260923 23:14:01.575977 5657 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:01.575997 5657 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:01.576025 5657 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:01.576499 5657 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260923 23:14:01.576582 5657 leader_election.cc:302] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9a3caf93379c4e8e9526de61cc1cf495; no voters:
I20260923 23:14:01.576716 5657 leader_election.cc:288] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:01.576761 5660 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:01.576927 5660 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [term 1 LEADER]: Becoming Leader. State: Replica: 9a3caf93379c4e8e9526de61cc1cf495, State: Running, Role: LEADER
I20260923 23:14:01.576941 5657 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:01.577059 5660 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } }
I20260923 23:14:01.577436 5662 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9a3caf93379c4e8e9526de61cc1cf495. Latest consensus state: current_term: 1 leader_uuid: "9a3caf93379c4e8e9526de61cc1cf495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } } }
I20260923 23:14:01.577513 5662 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:01.577577 5661 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9a3caf93379c4e8e9526de61cc1cf495" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9a3caf93379c4e8e9526de61cc1cf495" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34809 } } }
I20260923 23:14:01.577653 5661 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:01.577795 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5589
I20260923 23:14:01.577904 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:01.578109 16874 external_mini_cluster.cc:1428] Setting key fa17ccbbd60ad53852863ac5ca2943bf
I20260923 23:14:01.578343 5669 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:01.578843 5669 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:01.581087 5669 catalog_manager.cc:1347] Generated new cluster ID: 0568c25e9dab405e94f3846d4e6269f9
I20260923 23:14:01.581136 5669 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5576](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205241, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260923 23:14:01.589321 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:01.578692 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:54143 (local address 127.0.0.1:34809)
0923 23:14:01.579014 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:01.579023 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:01.579075 (+ 52us) server_negotiation.cc:407] Connection header received
0923 23:14:01.579193 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:01.579201 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:01.579267 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:01.579407 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:01.580119 (+ 712us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:01.581143 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:01.582212 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:01.582804 (+ 592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:01.582897 (+ 93us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:01.585732 (+ 2835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:01.585736 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:01.585736 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:01.585749 (+ 13us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:01.587654 (+ 1905us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:01.588100 (+ 446us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:01.588103 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:01.588103 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:01.588216 (+ 113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:01.588438 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:01.588438 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:01.588438 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:01.588742 (+ 304us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:01.588885 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:01.589003 (+ 118us) server_negotiation.cc:299] Negotiation successful
0923 23:14:01.589098 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":273,"thread_start_us":85,"threads_started":1}
I20260923 23:14:01.590320 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:01.598882 5679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:01.591885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:47147 (local address 127.0.0.1:34809)
0923 23:14:01.591923 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:01.591925 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:01.592047 (+ 122us) server_negotiation.cc:407] Connection header received
0923 23:14:01.592105 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:01.592105 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:01.592116 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:01.592155 (+ 39us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:01.592758 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:01.593529 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:01.594535 (+ 1006us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:01.594834 (+ 299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:01.594888 (+ 54us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:01.596290 (+ 1402us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:01.596291 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:01.596292 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:01.596298 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:01.597738 (+ 1440us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:01.598124 (+ 386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:01.598125 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:01.598126 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:01.598174 (+ 48us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:01.598373 (+ 199us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:01.598374 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:01.598374 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:01.598483 (+ 109us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:01.598568 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:01.598725 (+ 157us) server_negotiation.cc:299] Negotiation successful
0923 23:14:01.598788 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":15,"server-negotiator.queue_time_us":19}
I20260923 23:14:01.602558 5669 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:01.603327 5669 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:01.606349 5669 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9a3caf93379c4e8e9526de61cc1cf495: Generated new TSK 0
I20260923 23:14:01.606531 5669 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:01.610076 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5589
W20260923 23:14:01.613903 5574 connection.cc:535] client connection to 127.0.0.1:34809 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34809): Connection reset by peer (error 104)
2026-09-23T23:14:01Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (1207 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5688](info): set up 2 sockets
Sep 23 23:14:01 dist-test-slave-95hh krb5kdc[5688](info): commencing operation
krb5kdc: starting...
W20260923 23:14:02.555179 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:02 dist-test-slave-95hh krb5kdc[5688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:14:02.666100 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43451
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38233
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43451
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:02.744920 5701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:02.745133 5701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:02.745164 5701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:02.746330 5701 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:02.746371 5701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:02.746384 5701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:02.746395 5701 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:02.746409 5701 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:02.747931 5701 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38233
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43451
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5701
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:02.748172 5701 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:02.748377 5701 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:02.752868 5707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:02.752925 5701 server_base.cc:1034] running on GCE node
W20260923 23:14:02.752876 5709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:02.752876 5706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:02.753266 5701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:02.753552 5701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:02.754703 5701 hybrid_clock.cc:648] HybridClock initialized: now 1790205242754689 us; error 29 us; skew 500 ppm
Sep 23 23:14:02 dist-test-slave-95hh krb5kdc[5688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205242, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:02.757089 5701 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:14:02.757565 5701 webserver.cc:460] Webserver started at http://127.0.0.1:36429/ using document root <none> and password file <none>
I20260923 23:14:02.757784 5701 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:02.757833 5701 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:02.757956 5701 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:02.758991 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121"
format_stamp: "Formatted at 2026-09-23 23:14:02 on dist-test-slave-95hh"
server_key: "978cae95ff2e9c587aea805a0cee074b"
server_key_iv: "6a9d7e1d28a875fad88d676817b7966f"
server_key_version: "encryptionkey@0"
I20260923 23:14:02.759382 5701 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121"
format_stamp: "Formatted at 2026-09-23 23:14:02 on dist-test-slave-95hh"
server_key: "978cae95ff2e9c587aea805a0cee074b"
server_key_iv: "6a9d7e1d28a875fad88d676817b7966f"
server_key_version: "encryptionkey@0"
I20260923 23:14:02.760587 5701 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:02.761240 5716 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:02.761389 5701 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:14:02.761471 5701 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121"
format_stamp: "Formatted at 2026-09-23 23:14:02 on dist-test-slave-95hh"
server_key: "978cae95ff2e9c587aea805a0cee074b"
server_key_iv: "6a9d7e1d28a875fad88d676817b7966f"
server_key_version: "encryptionkey@0"
I20260923 23:14:02.761539 5701 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:02.788074 5701 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:02.789918 5701 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:02.790060 5701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:02.793764 5701 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43451
I20260923 23:14:02.793802 5768 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43451 every 8 connection(s)
I20260923 23:14:02.794138 5701 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:02.794973 5769 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:02.797251 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121: Bootstrap starting.
I20260923 23:14:02.797791 5769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:02.798036 5769 log.cc:824] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:02.798663 5769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121: No bootstrap required, opened a new log
I20260923 23:14:02.800282 5769 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } }
I20260923 23:14:02.800421 5769 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:02.800452 5769 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3a883d5a73a541ddb7d5f0d9ee6b4121, State: Initialized, Role: FOLLOWER
I20260923 23:14:02.800594 5769 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } }
I20260923 23:14:02.800678 5769 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:02.800715 5769 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:02.800762 5769 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:02.800820 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5701
I20260923 23:14:02.800932 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:02.801128 16874 external_mini_cluster.cc:1428] Setting key bda684bfd504b67250c0aa7026c42d61
I20260923 23:14:02.801411 5769 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } }
I20260923 23:14:02.801506 5769 leader_election.cc:302] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3a883d5a73a541ddb7d5f0d9ee6b4121; no voters:
I20260923 23:14:02.801663 5769 leader_election.cc:288] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:02.801813 5775 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:02.801918 5775 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [term 1 LEADER]: Becoming Leader. State: Replica: 3a883d5a73a541ddb7d5f0d9ee6b4121, State: Running, Role: LEADER
I20260923 23:14:02.802011 5775 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } }
I20260923 23:14:02.801914 5769 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:02.802508 5773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3a883d5a73a541ddb7d5f0d9ee6b4121. Latest consensus state: current_term: 1 leader_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } } }
I20260923 23:14:02.802619 5773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:02.802888 5778 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:02.803416 5778 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:02.804598 5776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3a883d5a73a541ddb7d5f0d9ee6b4121" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43451 } } }
I20260923 23:14:02.804740 5776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:02.804889 5778 catalog_manager.cc:1347] Generated new cluster ID: 8de4eb3abc804a5e9f13f61cb2972e13
I20260923 23:14:02.804929 5778 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:02.807093 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:02.801652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:43995 (local address 127.0.0.1:43451)
0923 23:14:02.801811 (+ 159us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:02.801821 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:02.801858 (+ 37us) server_negotiation.cc:407] Connection header received
0923 23:14:02.801979 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:02.801986 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:02.802046 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:02.802195 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:02.802400 (+ 205us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:02.803807 (+ 1407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:02.804996 (+ 1189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:02.805578 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:02.805694 (+ 116us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:02.806398 (+ 704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:02.806402 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:02.806402 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:02.806595 (+ 193us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:02.806639 (+ 44us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:02.806653 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:02.806729 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:02.806827 (+ 98us) server_negotiation.cc:299] Negotiation successful
0923 23:14:02.806853 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":6,"server-negotiator.queue_time_us":100,"thread_start_us":63,"threads_started":1}
I20260923 23:14:02.808010 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:02.813253 5774 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:02.809470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:41539 (local address 127.0.0.1:43451)
0923 23:14:02.809501 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:02.809503 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:02.809638 (+ 135us) server_negotiation.cc:407] Connection header received
0923 23:14:02.809703 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:02.809703 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:02.809712 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:02.809760 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:02.809899 (+ 139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:02.810920 (+ 1021us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:02.811946 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:02.812262 (+ 316us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:02.812320 (+ 58us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:02.812997 (+ 677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:02.812998 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:02.812999 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:02.813003 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:02.813017 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:02.813022 (+ 5us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:02.813072 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:02.813163 (+ 91us) server_negotiation.cc:299] Negotiation successful
0923 23:14:02.813176 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 23:14:02.821012 5778 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:02.821547 5778 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:02.825704 5778 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3a883d5a73a541ddb7d5f0d9ee6b4121: Generated new TSK 0
I20260923 23:14:02.825866 5778 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:02.829968 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5701
W20260923 23:14:02.833865 5686 connection.cc:535] client connection to 127.0.0.1:43451 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43451): Connection reset by peer (error 104)
2026-09-23T23:14:02Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (1220 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:02 dist-test-slave-95hh krb5kdc[5800](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:02 dist-test-slave-95hh krb5kdc[5800](info): set up 2 sockets
Sep 23 23:14:02 dist-test-slave-95hh krb5kdc[5800](info): commencing operation
krb5kdc: starting...
W20260923 23:14:03.774916 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.928s user 0.003s sys 0.002s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:03 dist-test-slave-95hh krb5kdc[5800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205243, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260923 23:14:03.886447 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44657
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:35213
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:03.968542 5813 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:03.968763 5813 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:03.968796 5813 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:03.969964 5813 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:03.970005 5813 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:03.970024 5813 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:03.970041 5813 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:03.970057 5813 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:03.971571 5813 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44657
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:35213
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:35213
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5813
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:03.971818 5813 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:03.972028 5813 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:03.976742 5821 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:03.976929 5813 server_base.cc:1034] running on GCE node
W20260923 23:14:03.976733 5819 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:03.976733 5818 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:03.977276 5813 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:03.977586 5813 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:03.978715 5813 hybrid_clock.cc:648] HybridClock initialized: now 1790205243978704 us; error 34 us; skew 500 ppm
Sep 23 23:14:03 dist-test-slave-95hh krb5kdc[5800](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:03.981128 5813 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260923 23:14:03.981590 5813 webserver.cc:460] Webserver started at http://127.0.0.1:32879/ using document root <none> and password file <none>
I20260923 23:14:03.981786 5813 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:03.981825 5813 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:03.981915 5813 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:03.982931 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "331ec72049cd47b29d23b61078748988"
format_stamp: "Formatted at 2026-09-23 23:14:03 on dist-test-slave-95hh"
server_key: "29193087c7e8c3d23f2d1f4533585af1"
server_key_iv: "ca95e6de08ad0d0047551713abfa6f3b"
server_key_version: "encryptionkey@0"
I20260923 23:14:03.983299 5813 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "331ec72049cd47b29d23b61078748988"
format_stamp: "Formatted at 2026-09-23 23:14:03 on dist-test-slave-95hh"
server_key: "29193087c7e8c3d23f2d1f4533585af1"
server_key_iv: "ca95e6de08ad0d0047551713abfa6f3b"
server_key_version: "encryptionkey@0"
I20260923 23:14:03.984488 5813 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:03.985214 5828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:03.985407 5813 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:03.985491 5813 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "331ec72049cd47b29d23b61078748988"
format_stamp: "Formatted at 2026-09-23 23:14:03 on dist-test-slave-95hh"
server_key: "29193087c7e8c3d23f2d1f4533585af1"
server_key_iv: "ca95e6de08ad0d0047551713abfa6f3b"
server_key_version: "encryptionkey@0"
I20260923 23:14:03.985548 5813 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:03.988770 5813 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:03.990742 5813 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:03.990845 5813 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:03.994479 5813 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:35213
I20260923 23:14:03.994520 5880 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:35213 every 8 connection(s)
I20260923 23:14:03.994838 5813 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:03.995728 5881 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:03.998039 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: Bootstrap starting.
I20260923 23:14:03.998615 5881 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:03.998867 5881 log.cc:824] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:03.999647 5881 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: No bootstrap required, opened a new log
I20260923 23:14:04.000996 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5813
I20260923 23:14:04.001124 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:04.001324 16874 external_mini_cluster.cc:1428] Setting key 03331aadedc2e9f81507356f197270db
I20260923 23:14:04.001540 5881 raft_consensus.cc:348] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260923 23:14:04.001689 5881 raft_consensus.cc:374] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:04.001719 5881 raft_consensus.cc:729] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 331ec72049cd47b29d23b61078748988, State: Initialized, Role: FOLLOWER
I20260923 23:14:04.002126 5881 consensus_queue.cc:260] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260923 23:14:04.002213 5881 raft_consensus.cc:388] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:04.002245 5881 raft_consensus.cc:482] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:04.002295 5881 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:04.002990 5881 raft_consensus.cc:504] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260923 23:14:04.003130 5881 leader_election.cc:302] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 331ec72049cd47b29d23b61078748988; no voters:
I20260923 23:14:04.003324 5881 leader_election.cc:288] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:04.003659 5881 sys_catalog.cc:564] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:04.003674 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:04.001893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:59041 (local address 127.0.0.1:35213)
0923 23:14:04.002127 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:04.002141 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:04.002176 (+ 35us) server_negotiation.cc:407] Connection header received
0923 23:14:04.002284 (+ 108us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:04.002291 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:04.002324 (+ 33us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:04.002455 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:04.003214 (+ 759us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:04.003218 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:04.003219 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:04.003223 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:04.003269 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:04.003284 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:04.003345 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:04.003432 (+ 87us) server_negotiation.cc:299] Negotiation successful
0923 23:14:04.003494 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":170,"thread_start_us":102,"threads_started":1}
I20260923 23:14:04.004554 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:04.003345 5886 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:04.004928 5886 raft_consensus.cc:686] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [term 1 LEADER]: Becoming Leader. State: Replica: 331ec72049cd47b29d23b61078748988, State: Running, Role: LEADER
I20260923 23:14:04.005045 5886 consensus_queue.cc:237] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } }
I20260923 23:14:04.005443 5898 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 331ec72049cd47b29d23b61078748988. Latest consensus state: current_term: 1 leader_uuid: "331ec72049cd47b29d23b61078748988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260923 23:14:04.005539 5898 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:04.005578 5887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "331ec72049cd47b29d23b61078748988" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "331ec72049cd47b29d23b61078748988" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 35213 } } }
I20260923 23:14:04.005661 5887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988 [sys.catalog]: This master's current role is: LEADER
W20260923 23:14:04.005996 5901 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:04.006132 5901 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:04.006232 5905 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:04.006471 5905 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:04.007233 5885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:04.006102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.145:38391 (local address 127.0.0.1:35213)
0923 23:14:04.006139 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:04.006141 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:04.006243 (+ 102us) server_negotiation.cc:407] Connection header received
0923 23:14:04.006331 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:04.006332 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:04.006337 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:04.006365 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:04.007001 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:04.007003 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:04.007003 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:04.007007 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:04.007025 (+ 18us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:04.007053 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:04.007114 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:04.007160 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 23:14:04.007176 (+ 16us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:04.007980 5905 catalog_manager.cc:1347] Generated new cluster ID: 9fbbe0c0c4f04015afa577ac3777f6b8
I20260923 23:14:04.008031 5905 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:04.034619 5905 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:04.034801 5905 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:04.038163 5905 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 331ec72049cd47b29d23b61078748988: Generated new TSK 0
I20260923 23:14:04.038335 5905 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:04.046387 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5813
2026-09-23T23:14:04Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (1216 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:04 dist-test-slave-95hh krb5kdc[5912](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:04 dist-test-slave-95hh krb5kdc[5912](info): set up 2 sockets
Sep 23 23:14:04 dist-test-slave-95hh krb5kdc[5912](info): commencing operation
krb5kdc: starting...
W20260923 23:14:04.994915 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.931s user 0.005s sys 0.000s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[5912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:05Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:05.109647 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37935
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:43869
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:37935
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:05.189595 5925 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:05.189826 5925 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:05.189849 5925 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:05.191134 5925 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:05.191183 5925 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:05.191196 5925 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:05.191207 5925 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:05.191217 5925 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:05.192767 5925 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:43869
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:37935
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:37935
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.5925
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:05.193012 5925 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:05.193230 5925 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:05.197957 5933 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:05.198019 5925 server_base.cc:1034] running on GCE node
W20260923 23:14:05.197950 5931 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:05.197950 5930 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:05.198395 5925 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:05.198684 5925 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:05.199851 5925 hybrid_clock.cc:648] HybridClock initialized: now 1790205245199832 us; error 29 us; skew 500 ppm
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[5912](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205245, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:05.206840 5925 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:05.207427 5925 webserver.cc:460] Webserver started at http://127.16.122.190:34625/ using document root <none> and password file <none>
I20260923 23:14:05.207660 5925 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:05.207708 5925 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:05.207830 5925 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:05.208927 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "94042df3c9ec4d47acd7d2136dfe4f57"
format_stamp: "Formatted at 2026-09-23 23:14:05 on dist-test-slave-95hh"
server_key: "191588ed715c09be76f6fca3d109e04b"
server_key_iv: "e07f461adec54238ccdde09b463148cc"
server_key_version: "encryptionkey@0"
I20260923 23:14:05.209254 5925 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "94042df3c9ec4d47acd7d2136dfe4f57"
format_stamp: "Formatted at 2026-09-23 23:14:05 on dist-test-slave-95hh"
server_key: "191588ed715c09be76f6fca3d109e04b"
server_key_iv: "e07f461adec54238ccdde09b463148cc"
server_key_version: "encryptionkey@0"
I20260923 23:14:05.210577 5925 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:14:05.211385 5940 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:05.211577 5925 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:14:05.211665 5925 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "94042df3c9ec4d47acd7d2136dfe4f57"
format_stamp: "Formatted at 2026-09-23 23:14:05 on dist-test-slave-95hh"
server_key: "191588ed715c09be76f6fca3d109e04b"
server_key_iv: "e07f461adec54238ccdde09b463148cc"
server_key_version: "encryptionkey@0"
I20260923 23:14:05.211723 5925 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:05.230032 5925 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:05.231886 5925 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:05.232025 5925 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:05.235734 5925 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:37935
I20260923 23:14:05.235769 5992 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:37935 every 8 connection(s)
I20260923 23:14:05.236105 5925 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:05.237126 5993 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:05.240067 5993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57: Bootstrap starting.
I20260923 23:14:05.240752 5993 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:05.241042 5993 log.cc:824] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:05.241894 5993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57: No bootstrap required, opened a new log
I20260923 23:14:05.243678 5993 raft_consensus.cc:348] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } }
I20260923 23:14:05.243847 5993 raft_consensus.cc:374] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:05.243872 5993 raft_consensus.cc:729] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94042df3c9ec4d47acd7d2136dfe4f57, State: Initialized, Role: FOLLOWER
I20260923 23:14:05.244021 5993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } }
I20260923 23:14:05.244102 5993 raft_consensus.cc:388] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:05.244127 5993 raft_consensus.cc:482] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:05.244165 5993 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:05.244434 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 5925
I20260923 23:14:05.244567 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:05.244737 16874 external_mini_cluster.cc:1428] Setting key 333fa2c75b7623945cdcd689fb23ca61
I20260923 23:14:05.245288 5993 raft_consensus.cc:504] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } }
I20260923 23:14:05.245416 5993 leader_election.cc:302] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94042df3c9ec4d47acd7d2136dfe4f57; no voters:
I20260923 23:14:05.245584 5993 leader_election.cc:288] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:05.245616 5998 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:05.245712 5998 raft_consensus.cc:686] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [term 1 LEADER]: Becoming Leader. State: Replica: 94042df3c9ec4d47acd7d2136dfe4f57, State: Running, Role: LEADER
I20260923 23:14:05.245848 5998 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } }
I20260923 23:14:05.245858 5993 sys_catalog.cc:564] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:05.246647 5999 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } } }
I20260923 23:14:05.246995 6000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94042df3c9ec4d47acd7d2136dfe4f57. Latest consensus state: current_term: 1 leader_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94042df3c9ec4d47acd7d2136dfe4f57" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 37935 } } }
I20260923 23:14:05.247130 5999 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:05.247205 6000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:05.247373 6011 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:05.247821 6011 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:05.249850 6011 catalog_manager.cc:1347] Generated new cluster ID: af73cbeaca67449f8885a41e2665603d
I20260923 23:14:05.249948 6011 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[5912](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205245, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:05.256238 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:05.245298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45749 (local address 127.16.122.190:37935)
0923 23:14:05.245492 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:05.245503 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:05.245537 (+ 34us) server_negotiation.cc:407] Connection header received
0923 23:14:05.245644 (+ 107us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:05.245652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:05.245713 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:05.245882 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:05.246615 (+ 733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:05.248090 (+ 1475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:05.248975 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:05.249541 (+ 566us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:05.249647 (+ 106us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:05.252496 (+ 2849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:05.252501 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:05.252501 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:05.252515 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:05.254430 (+ 1915us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:05.254902 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:05.254905 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:05.254906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:05.255056 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:05.255281 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:05.255282 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:05.255282 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:05.255615 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:05.255763 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:05.255909 (+ 146us) server_negotiation.cc:299] Negotiation successful
0923 23:14:05.256011 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138,"thread_start_us":91,"threads_started":1}
I20260923 23:14:05.257299 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:05.266167 5997 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:05.258890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41039 (local address 127.16.122.190:37935)
0923 23:14:05.258937 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:05.258939 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:05.259173 (+ 234us) server_negotiation.cc:407] Connection header received
0923 23:14:05.259221 (+ 48us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:05.259221 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:05.259231 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:05.259297 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:05.259879 (+ 582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:05.260653 (+ 774us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:05.261604 (+ 951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:05.261942 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:05.262010 (+ 68us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:05.263479 (+ 1469us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:05.263481 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:05.263482 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:05.263490 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:05.265021 (+ 1531us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:05.265417 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:05.265418 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:05.265418 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:05.265477 (+ 59us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:05.265664 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:05.265665 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:05.265665 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:05.265799 (+ 134us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:05.265900 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:05.266004 (+ 104us) server_negotiation.cc:299] Negotiation successful
0923 23:14:05.266066 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:14:05.273306 6011 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:05.274066 6011 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:05.286984 6011 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 94042df3c9ec4d47acd7d2136dfe4f57: Generated new TSK 0
I20260923 23:14:05.287247 6011 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:05.294672 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 5925
W20260923 23:14:05.298501 5910 connection.cc:535] client connection to 127.16.122.190:37935 recv error: Network error: failed to read from TLS socket (remote: 127.16.122.190:37935): Connection reset by peer (error 104)
2026-09-23T23:14:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (1248 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[6024](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[6024](info): set up 2 sockets
Sep 23 23:14:05 dist-test-slave-95hh krb5kdc[6024](info): commencing operation
krb5kdc: starting...
W20260923 23:14:06.239015 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.927s user 0.002s sys 0.003s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:06 dist-test-slave-95hh krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:06.349658 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34675
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:40265
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34675
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:06.428833 6037 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:06.429044 6037 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:06.429065 6037 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:06.430187 6037 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:06.430227 6037 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:06.430239 6037 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:06.430262 6037 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:06.430280 6037 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:06.431772 6037 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:40265
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34675
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34675
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6037
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:06.432006 6037 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:06.432204 6037 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:06.436697 6045 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:06.436693 6043 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:06.436677 6042 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:06.436751 6037 server_base.cc:1034] running on GCE node
I20260923 23:14:06.437151 6037 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:06.437433 6037 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:06.438570 6037 hybrid_clock.cc:648] HybridClock initialized: now 1790205246438556 us; error 30 us; skew 500 ppm
Sep 23 23:14:06 dist-test-slave-95hh krb5kdc[6024](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205246, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:06.440864 6037 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:06.441267 6037 webserver.cc:460] Webserver started at http://127.16.122.190:33015/ using document root <none> and password file <none>
I20260923 23:14:06.441443 6037 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:06.441480 6037 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:06.441569 6037 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:06.442539 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "265aaa5c0c2545af919c8890d6fbebb0"
format_stamp: "Formatted at 2026-09-23 23:14:06 on dist-test-slave-95hh"
server_key: "44531b3e609c58da1a6c76595c9825a5"
server_key_iv: "d83a581aae016b0ffca515fd62a9f560"
server_key_version: "encryptionkey@0"
I20260923 23:14:06.442827 6037 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "265aaa5c0c2545af919c8890d6fbebb0"
format_stamp: "Formatted at 2026-09-23 23:14:06 on dist-test-slave-95hh"
server_key: "44531b3e609c58da1a6c76595c9825a5"
server_key_iv: "d83a581aae016b0ffca515fd62a9f560"
server_key_version: "encryptionkey@0"
I20260923 23:14:06.444103 6037 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:14:06.444710 6052 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:06.444864 6037 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:06.444947 6037 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "265aaa5c0c2545af919c8890d6fbebb0"
format_stamp: "Formatted at 2026-09-23 23:14:06 on dist-test-slave-95hh"
server_key: "44531b3e609c58da1a6c76595c9825a5"
server_key_iv: "d83a581aae016b0ffca515fd62a9f560"
server_key_version: "encryptionkey@0"
I20260923 23:14:06.445000 6037 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:06.455865 6037 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:06.457594 6037 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:06.457700 6037 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:06.461277 6037 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34675
I20260923 23:14:06.461308 6104 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34675 every 8 connection(s)
I20260923 23:14:06.461617 6037 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:06.462428 6105 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:06.464154 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6037
I20260923 23:14:06.464248 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:06.464452 16874 external_mini_cluster.cc:1428] Setting key 6e7931144ab672f030465c7376b20f8f
I20260923 23:14:06.464882 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0: Bootstrap starting.
I20260923 23:14:06.465582 6105 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:06.465883 6105 log.cc:824] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:06.466571 6105 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0: No bootstrap required, opened a new log
I20260923 23:14:06.468736 6105 raft_consensus.cc:348] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } }
I20260923 23:14:06.468904 6105 raft_consensus.cc:374] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:06.468951 6105 raft_consensus.cc:729] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 265aaa5c0c2545af919c8890d6fbebb0, State: Initialized, Role: FOLLOWER
I20260923 23:14:06.469080 6105 consensus_queue.cc:260] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } }
I20260923 23:14:06.469151 6105 raft_consensus.cc:388] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:06.469182 6105 raft_consensus.cc:482] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:06.469236 6105 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:06.469763 6105 raft_consensus.cc:504] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } }
I20260923 23:14:06.469884 6105 leader_election.cc:302] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 265aaa5c0c2545af919c8890d6fbebb0; no voters:
I20260923 23:14:06.469907 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:06.464932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43831 (local address 127.16.122.190:34675)
0923 23:14:06.465096 (+ 164us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:06.465105 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:06.465137 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:14:06.465686 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:06.465696 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:06.465765 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:06.465924 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:06.466103 (+ 179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:06.467087 (+ 984us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:06.468145 (+ 1058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:06.468528 (+ 383us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:06.468598 (+ 70us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:06.469303 (+ 705us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:06.469307 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:06.469307 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:06.469472 (+ 165us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:06.469502 (+ 30us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:06.469513 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:06.469581 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:06.469686 (+ 105us) server_negotiation.cc:299] Negotiation successful
0923 23:14:06.469713 (+ 27us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113,"thread_start_us":75,"threads_started":1}
I20260923 23:14:06.470037 6105 leader_election.cc:288] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:06.470180 6111 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:06.470265 6105 sys_catalog.cc:564] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:06.470311 6111 raft_consensus.cc:686] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [term 1 LEADER]: Becoming Leader. State: Replica: 265aaa5c0c2545af919c8890d6fbebb0, State: Running, Role: LEADER
I20260923 23:14:06.470384 6111 consensus_queue.cc:237] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } }
I20260923 23:14:06.470748 6112 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 265aaa5c0c2545af919c8890d6fbebb0. Latest consensus state: current_term: 1 leader_uuid: "265aaa5c0c2545af919c8890d6fbebb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } } }
I20260923 23:14:06.470760 6110 sys_catalog.cc:455] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "265aaa5c0c2545af919c8890d6fbebb0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "265aaa5c0c2545af919c8890d6fbebb0" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34675 } } }
I20260923 23:14:06.470837 6112 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:06.470865 6110 sys_catalog.cc:458] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:06.470873 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:06.471057 6119 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:06.471534 6119 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:06.474905 6119 catalog_manager.cc:1347] Generated new cluster ID: e95e1cf9192c4241999309a41d909ae7
I20260923 23:14:06.474979 6119 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:06.475786 6108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:06.472336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33423 (local address 127.16.122.190:34675)
0923 23:14:06.472371 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:06.472372 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:06.472506 (+ 134us) server_negotiation.cc:407] Connection header received
0923 23:14:06.472551 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:06.472552 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:06.472563 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:06.472619 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:06.472772 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:06.473464 (+ 692us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:06.474506 (+ 1042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:06.474741 (+ 235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:06.474803 (+ 62us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:06.475566 (+ 763us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:06.475567 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:06.475568 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:06.475572 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:06.475588 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:06.475592 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:06.475637 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:06.475703 (+ 66us) server_negotiation.cc:299] Negotiation successful
0923 23:14:06.475712 (+ 9us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:14:06.485532 6119 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:06.486246 6119 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:06.492883 6119 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 265aaa5c0c2545af919c8890d6fbebb0: Generated new TSK 0
I20260923 23:14:06.493074 6119 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:06.496255 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6037
W20260923 23:14:06.499802 6022 connection.cc:535] client connection to 127.16.122.190:34675 recv error: Network error: failed to read from TLS socket (remote: 127.16.122.190:34675): Connection reset by peer (error 104)
2026-09-23T23:14:06Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (1201 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:06 dist-test-slave-95hh krb5kdc[6136](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:06 dist-test-slave-95hh krb5kdc[6136](info): set up 2 sockets
Sep 23 23:14:06 dist-test-slave-95hh krb5kdc[6136](info): commencing operation
krb5kdc: starting...
W20260923 23:14:07.442816 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.930s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:07 dist-test-slave-95hh krb5kdc[6136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205247, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:07Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:07.553418 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:46365
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:42865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:46365
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:07.632972 6149 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:07.633188 6149 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:07.633210 6149 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:07.634336 6149 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:07.634377 6149 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:07.634389 6149 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:07.634402 6149 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:07.634425 6149 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:07.635910 6149 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:42865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:46365
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:46365
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6149
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:07.636135 6149 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:07.636343 6149 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:07.641182 6157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:07.641193 6149 server_base.cc:1034] running on GCE node
W20260923 23:14:07.641171 6155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:07.641175 6154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:07.641716 6149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:07.641993 6149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:07.643098 6149 hybrid_clock.cc:648] HybridClock initialized: now 1790205247643083 us; error 28 us; skew 500 ppm
Sep 23 23:14:07 dist-test-slave-95hh krb5kdc[6136](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205247, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:07.645503 6149 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:07.645925 6149 webserver.cc:460] Webserver started at http://127.16.122.190:36527/ using document root <none> and password file <none>
I20260923 23:14:07.646102 6149 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:07.646142 6149 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:07.646236 6149 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:07.647310 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "3174a881921541a7bdf067f5aa6fa61e"
format_stamp: "Formatted at 2026-09-23 23:14:07 on dist-test-slave-95hh"
server_key: "8d617bcf1faa06f74a0a12f5326780b5"
server_key_iv: "68c6e8d6ac6ff5e7b14b2eff44fcb5ba"
server_key_version: "encryptionkey@0"
I20260923 23:14:07.647610 6149 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "3174a881921541a7bdf067f5aa6fa61e"
format_stamp: "Formatted at 2026-09-23 23:14:07 on dist-test-slave-95hh"
server_key: "8d617bcf1faa06f74a0a12f5326780b5"
server_key_iv: "68c6e8d6ac6ff5e7b14b2eff44fcb5ba"
server_key_version: "encryptionkey@0"
I20260923 23:14:07.648749 6149 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:07.649390 6164 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:07.649530 6149 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:14:07.649606 6149 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "3174a881921541a7bdf067f5aa6fa61e"
format_stamp: "Formatted at 2026-09-23 23:14:07 on dist-test-slave-95hh"
server_key: "8d617bcf1faa06f74a0a12f5326780b5"
server_key_iv: "68c6e8d6ac6ff5e7b14b2eff44fcb5ba"
server_key_version: "encryptionkey@0"
I20260923 23:14:07.649672 6149 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:07.652814 6149 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:07.654829 6149 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:07.654951 6149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:07.658581 6149 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:46365
I20260923 23:14:07.658615 6216 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:46365 every 8 connection(s)
I20260923 23:14:07.658914 6149 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:07.659873 6217 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:07.662091 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e: Bootstrap starting.
I20260923 23:14:07.662633 6217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:07.662904 6217 log.cc:824] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:07.663547 6217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e: No bootstrap required, opened a new log
I20260923 23:14:07.665084 6217 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } }
I20260923 23:14:07.665220 6217 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:07.665251 6217 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3174a881921541a7bdf067f5aa6fa61e, State: Initialized, Role: FOLLOWER
I20260923 23:14:07.665385 6217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } }
I20260923 23:14:07.665444 6217 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:07.665473 6217 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:07.665519 6217 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:07.666301 6217 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } }
I20260923 23:14:07.666394 6217 leader_election.cc:302] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3174a881921541a7bdf067f5aa6fa61e; no voters:
I20260923 23:14:07.666550 6217 leader_election.cc:288] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:07.666579 6220 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:07.666710 6220 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [term 1 LEADER]: Becoming Leader. State: Replica: 3174a881921541a7bdf067f5aa6fa61e, State: Running, Role: LEADER
I20260923 23:14:07.666791 6217 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:07.666805 6220 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } }
I20260923 23:14:07.667176 6221 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3174a881921541a7bdf067f5aa6fa61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } } }
I20260923 23:14:07.667254 6221 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:07.667204 6222 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3174a881921541a7bdf067f5aa6fa61e. Latest consensus state: current_term: 1 leader_uuid: "3174a881921541a7bdf067f5aa6fa61e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3174a881921541a7bdf067f5aa6fa61e" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 46365 } } }
I20260923 23:14:07.667339 6222 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:07.667547 6226 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:07.667814 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6149
I20260923 23:14:07.667913 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:07.668063 6226 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:07.668098 16874 external_mini_cluster.cc:1428] Setting key a74b51e535802cdd602038df184daa9f
I20260923 23:14:07.669550 6226 catalog_manager.cc:1347] Generated new cluster ID: 3f52586d3d25426ab049ef84e8f2f50f
I20260923 23:14:07.669601 6226 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:07.670243 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:07.668678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36345 (local address 127.16.122.190:46365)
0923 23:14:07.668836 (+ 158us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:07.668844 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:07.668879 (+ 35us) server_negotiation.cc:407] Connection header received
0923 23:14:07.668994 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:07.669001 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:07.669031 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:07.669158 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:07.669828 (+ 670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:07.669832 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:07.669832 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:07.669837 (+ 5us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:07.669878 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:07.669890 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:07.669945 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:07.670021 (+ 76us) server_negotiation.cc:299] Negotiation successful
0923 23:14:07.670062 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":80,"threads_started":1}
I20260923 23:14:07.671016 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:07.673341 6239 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:07.672398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35879 (local address 127.16.122.190:46365)
0923 23:14:07.672432 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:07.672433 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:07.672531 (+ 98us) server_negotiation.cc:407] Connection header received
0923 23:14:07.672584 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:07.672585 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:07.672590 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:07.672634 (+ 44us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:07.673151 (+ 517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:07.673152 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:07.673152 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:07.673156 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:07.673170 (+ 14us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:07.673174 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:07.673227 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:07.673269 (+ 42us) server_negotiation.cc:299] Negotiation successful
0923 23:14:07.673282 (+ 13us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:07.696029 6226 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:07.696244 6226 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:07.705096 6226 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3174a881921541a7bdf067f5aa6fa61e: Generated new TSK 0
I20260923 23:14:07.705319 6226 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:07.711314 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6149
2026-09-23T23:14:07Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (1215 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:07 dist-test-slave-95hh krb5kdc[6248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:07 dist-test-slave-95hh krb5kdc[6248](info): set up 2 sockets
Sep 23 23:14:07 dist-test-slave-95hh krb5kdc[6248](info): commencing operation
krb5kdc: starting...
W20260923 23:14:08.666872 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.938s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:08.773723 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34807
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:37743
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34807
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:08.852969 6261 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:08.853197 6261 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:08.853229 6261 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:08.854466 6261 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:08.854511 6261 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:08.854530 6261 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:08.854547 6261 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:08.854561 6261 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:08.856046 6261 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:37743
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6261
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:08.856297 6261 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:08.856503 6261 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:08.861282 6269 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:08.861254 6267 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:08.861332 6266 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:08.861426 6261 server_base.cc:1034] running on GCE node
I20260923 23:14:08.861737 6261 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:08.862013 6261 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:08.863150 6261 hybrid_clock.cc:648] HybridClock initialized: now 1790205248863136 us; error 29 us; skew 500 ppm
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205248, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:08.865510 6261 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:08.865947 6261 webserver.cc:460] Webserver started at http://127.16.122.190:35577/ using document root <none> and password file <none>
I20260923 23:14:08.866133 6261 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:08.866173 6261 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:08.866272 6261 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:08.867345 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "34f1ab19111b447da58cc617b525eb51"
format_stamp: "Formatted at 2026-09-23 23:14:08 on dist-test-slave-95hh"
server_key: "ea8fef84b931f6c161d32fca292e822e"
server_key_iv: "55b6c8cdfbaec4e9548af9e3b4ca1a98"
server_key_version: "encryptionkey@0"
I20260923 23:14:08.867671 6261 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "34f1ab19111b447da58cc617b525eb51"
format_stamp: "Formatted at 2026-09-23 23:14:08 on dist-test-slave-95hh"
server_key: "ea8fef84b931f6c161d32fca292e822e"
server_key_iv: "55b6c8cdfbaec4e9548af9e3b4ca1a98"
server_key_version: "encryptionkey@0"
I20260923 23:14:08.868916 6261 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.002s sys 0.000s
I20260923 23:14:08.869606 6276 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:08.869755 6261 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:14:08.869832 6261 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "34f1ab19111b447da58cc617b525eb51"
format_stamp: "Formatted at 2026-09-23 23:14:08 on dist-test-slave-95hh"
server_key: "ea8fef84b931f6c161d32fca292e822e"
server_key_iv: "55b6c8cdfbaec4e9548af9e3b4ca1a98"
server_key_version: "encryptionkey@0"
I20260923 23:14:08.869896 6261 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:08.881897 6261 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:08.884140 6261 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:08.884284 6261 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:08.888089 6261 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34807
I20260923 23:14:08.888131 6328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34807 every 8 connection(s)
I20260923 23:14:08.888460 6261 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:08.889285 6329 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:08.891597 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51: Bootstrap starting.
I20260923 23:14:08.892174 6329 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:08.892431 6329 log.cc:824] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:08.893085 6329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51: No bootstrap required, opened a new log
I20260923 23:14:08.894662 6329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } }
I20260923 23:14:08.894809 6329 raft_consensus.cc:374] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:08.894830 6329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 34f1ab19111b447da58cc617b525eb51, State: Initialized, Role: FOLLOWER
I20260923 23:14:08.894980 6329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } }
I20260923 23:14:08.895082 6329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:08.895108 6329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:08.895164 6329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:08.895994 6329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } }
I20260923 23:14:08.896093 6329 leader_election.cc:302] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 34f1ab19111b447da58cc617b525eb51; no voters:
I20260923 23:14:08.896263 6329 leader_election.cc:288] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:08.896307 6332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:08.896445 6332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [term 1 LEADER]: Becoming Leader. State: Replica: 34f1ab19111b447da58cc617b525eb51, State: Running, Role: LEADER
I20260923 23:14:08.896517 6329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:08.896539 6332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } }
I20260923 23:14:08.896909 6333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "34f1ab19111b447da58cc617b525eb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } } }
I20260923 23:14:08.896920 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 34f1ab19111b447da58cc617b525eb51. Latest consensus state: current_term: 1 leader_uuid: "34f1ab19111b447da58cc617b525eb51" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "34f1ab19111b447da58cc617b525eb51" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34807 } } }
I20260923 23:14:08.897017 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:08.897009 6333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:08.897279 6341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:08.897822 6341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:08.898030 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6261
I20260923 23:14:08.898138 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:08.898315 16874 external_mini_cluster.cc:1428] Setting key c0a5c5ae931bdceb4bf905e00304a804
I20260923 23:14:08.899416 6341 catalog_manager.cc:1347] Generated new cluster ID: 44781bfea3894008b56558de0224efc9
I20260923 23:14:08.899471 6341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:08.915791 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:08.898842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53521 (local address 127.16.122.190:34807)
0923 23:14:08.898985 (+ 143us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:08.898994 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:08.907357 (+ 8363us) server_negotiation.cc:407] Connection header received
0923 23:14:08.907507 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:08.907515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:08.907567 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:08.907693 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:08.908342 (+ 649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:08.909279 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:08.910003 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:08.910441 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:08.910538 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:08.912750 (+ 2212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:08.912753 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:08.912754 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:08.912764 (+ 10us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:08.914256 (+ 1492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:08.914571 (+ 315us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:08.914574 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:08.914574 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:08.914658 (+ 84us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:08.914830 (+ 172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:08.914830 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:08.914831 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:08.915109 (+ 278us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:08.915239 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:08.915363 (+ 124us) server_negotiation.cc:299] Negotiation successful
0923 23:14:08.915591 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99,"thread_start_us":73,"threads_started":1}
I20260923 23:14:08.916666 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:08.924742 6341 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:08.925487 6341 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:08.929257 6351 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:08.919280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36677 (local address 127.16.122.190:34807)
0923 23:14:08.919318 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:08.919320 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:08.923233 (+ 3913us) server_negotiation.cc:407] Connection header received
0923 23:14:08.923293 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:08.923294 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:08.923306 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:08.923356 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:08.923885 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:08.924619 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:08.925413 (+ 794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:08.925657 (+ 244us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:08.925720 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:08.926889 (+ 1169us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:08.926891 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:08.926891 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:08.926899 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:08.928176 (+ 1277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:08.928489 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:08.928490 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:08.928491 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:08.928546 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:08.928712 (+ 166us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:08.928713 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:08.928713 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:08.928873 (+ 160us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:08.928955 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:08.929076 (+ 121us) server_negotiation.cc:299] Negotiation successful
0923 23:14:08.929168 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:08.930006 6341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 34f1ab19111b447da58cc617b525eb51: Generated new TSK 0
I20260923 23:14:08.930189 6341 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:08.932734 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6261
2026-09-23T23:14:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (1221 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6360](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6360](info): set up 2 sockets
Sep 23 23:14:08 dist-test-slave-95hh krb5kdc[6360](info): commencing operation
krb5kdc: starting...
W20260923 23:14:09.870885 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.921s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:09 dist-test-slave-95hh krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205249, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:09Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:09Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:09.983650 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:33717
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:39755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:33717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:10.061738 6373 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:10.061952 6373 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:10.061975 6373 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:10.063119 6373 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:10.063154 6373 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:10.063166 6373 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:10.063179 6373 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:10.063189 6373 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:10.064595 6373 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:39755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:33717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:33717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6373
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:10.064788 6373 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:10.064968 6373 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:10.069408 6378 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:10.069485 6379 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:10.069412 6381 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:10.069605 6373 server_base.cc:1034] running on GCE node
I20260923 23:14:10.069829 6373 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:10.070117 6373 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:10.071249 6373 hybrid_clock.cc:648] HybridClock initialized: now 1790205250071230 us; error 36 us; skew 500 ppm
Sep 23 23:14:10 dist-test-slave-95hh krb5kdc[6360](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205250, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:10.073612 6373 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:10.073982 6373 webserver.cc:460] Webserver started at http://127.16.122.190:34633/ using document root <none> and password file <none>
I20260923 23:14:10.074158 6373 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:10.074195 6373 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:10.074282 6373 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:10.075330 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "2e96dc5cdaeb473eac2a469c26bb53e7"
format_stamp: "Formatted at 2026-09-23 23:14:10 on dist-test-slave-95hh"
server_key: "a23a85e1f3f00a3d3bd066beed08d93b"
server_key_iv: "4b5d19add71683d8f054572e92565685"
server_key_version: "encryptionkey@0"
I20260923 23:14:10.075621 6373 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "2e96dc5cdaeb473eac2a469c26bb53e7"
format_stamp: "Formatted at 2026-09-23 23:14:10 on dist-test-slave-95hh"
server_key: "a23a85e1f3f00a3d3bd066beed08d93b"
server_key_iv: "4b5d19add71683d8f054572e92565685"
server_key_version: "encryptionkey@0"
I20260923 23:14:10.076738 6373 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:10.077335 6388 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:10.077473 6373 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:14:10.077569 6373 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "2e96dc5cdaeb473eac2a469c26bb53e7"
format_stamp: "Formatted at 2026-09-23 23:14:10 on dist-test-slave-95hh"
server_key: "a23a85e1f3f00a3d3bd066beed08d93b"
server_key_iv: "4b5d19add71683d8f054572e92565685"
server_key_version: "encryptionkey@0"
I20260923 23:14:10.077628 6373 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:10.097980 6373 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:10.100260 6373 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:10.100400 6373 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:10.104146 6373 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:33717
I20260923 23:14:10.104202 6440 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:33717 every 8 connection(s)
I20260923 23:14:10.104523 6373 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:10.105362 6441 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:10.107642 6441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7: Bootstrap starting.
I20260923 23:14:10.108086 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6373
I20260923 23:14:10.108206 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:10.108284 6441 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:10.108402 16874 external_mini_cluster.cc:1428] Setting key 8810afcbd9da201711fa4c94c722f311
I20260923 23:14:10.108595 6441 log.cc:824] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:10.109294 6441 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7: No bootstrap required, opened a new log
I20260923 23:14:10.111621 6441 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } }
I20260923 23:14:10.111791 6441 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:10.111820 6441 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e96dc5cdaeb473eac2a469c26bb53e7, State: Initialized, Role: FOLLOWER
I20260923 23:14:10.111956 6441 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } }
I20260923 23:14:10.112040 6441 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:10.112078 6441 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:10.112118 6441 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:10.112666 6441 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } }
I20260923 23:14:10.112776 6441 leader_election.cc:302] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e96dc5cdaeb473eac2a469c26bb53e7; no voters:
I20260923 23:14:10.112947 6441 leader_election.cc:288] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:10.113093 6447 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:10.113188 6441 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:10.113291 6447 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [term 1 LEADER]: Becoming Leader. State: Replica: 2e96dc5cdaeb473eac2a469c26bb53e7, State: Running, Role: LEADER
I20260923 23:14:10.113421 6447 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } }
I20260923 23:14:10.114125 6448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e96dc5cdaeb473eac2a469c26bb53e7. Latest consensus state: current_term: 1 leader_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } } }
I20260923 23:14:10.114189 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:10.109048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44833 (local address 127.16.122.190:33717)
0923 23:14:10.109213 (+ 165us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:10.109223 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:10.109249 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:14:10.109722 (+ 473us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:10.109728 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:10.109779 (+ 51us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:10.109893 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:10.110101 (+ 208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:10.110949 (+ 848us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:10.112018 (+ 1069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:10.112553 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:10.112633 (+ 80us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:10.113390 (+ 757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:10.113393 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:10.113394 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:10.113605 (+ 211us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:10.113646 (+ 41us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:10.113662 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:10.113714 (+ 52us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:10.113827 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 23:14:10.113898 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114,"thread_start_us":73,"threads_started":1}
I20260923 23:14:10.114236 6448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:10.114131 6446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e96dc5cdaeb473eac2a469c26bb53e7" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 33717 } } }
I20260923 23:14:10.114408 6446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:10.114533 6455 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:10.114992 6455 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:10.115120 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:10.116860 6455 catalog_manager.cc:1347] Generated new cluster ID: feff16c5996b4cfdae6e4ad3c66a57ff
I20260923 23:14:10.116912 6455 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:10.125725 6445 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:10.116471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52063 (local address 127.16.122.190:33717)
0923 23:14:10.116508 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:10.116510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:10.116692 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:14:10.116735 (+ 43us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:10.116735 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:10.116745 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:10.116796 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:10.123331 (+ 6535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:10.123850 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:10.124577 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:10.124780 (+ 203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:10.124839 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:10.125475 (+ 636us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:10.125477 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:10.125477 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:10.125483 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:10.125500 (+ 17us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:10.125504 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:10.125549 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:10.125612 (+ 63us) server_negotiation.cc:299] Negotiation successful
0923 23:14:10.125638 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:14:10.149441 6455 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:10.149986 6455 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:10.155251 6455 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2e96dc5cdaeb473eac2a469c26bb53e7: Generated new TSK 0
I20260923 23:14:10.155452 6455 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:10.157333 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6373
2026-09-23T23:14:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (1224 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:10 dist-test-slave-95hh krb5kdc[6472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:10 dist-test-slave-95hh krb5kdc[6472](info): set up 2 sockets
Sep 23 23:14:10 dist-test-slave-95hh krb5kdc[6472](info): commencing operation
krb5kdc: starting...
W20260923 23:14:11.098861 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.924s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:11 dist-test-slave-95hh krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205251, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:11.208976 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34517
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:37641
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:34517
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:11.289402 6485 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:11.289620 6485 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:11.289650 6485 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:11.290814 6485 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:11.290860 6485 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:11.290879 6485 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:11.290894 6485 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:11.290910 6485 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:11.292444 6485 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:37641
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:34517
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:34517
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6485
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:11.292688 6485 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:11.292901 6485 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:11.297526 6490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:11.297665 6485 server_base.cc:1034] running on GCE node
W20260923 23:14:11.297508 6493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:11.297551 6491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:11.298008 6485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:11.298302 6485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:11.299436 6485 hybrid_clock.cc:648] HybridClock initialized: now 1790205251299421 us; error 32 us; skew 500 ppm
Sep 23 23:14:11 dist-test-slave-95hh krb5kdc[6472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205251, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:11.304733 6485 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:11.305224 6485 webserver.cc:460] Webserver started at http://127.16.122.190:35259/ using document root <none> and password file <none>
I20260923 23:14:11.305451 6485 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:11.305506 6485 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:11.305632 6485 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:11.306710 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "73a68c633a4e4b9395b784f7c5bb9920"
format_stamp: "Formatted at 2026-09-23 23:14:11 on dist-test-slave-95hh"
server_key: "f25568f272665f740c0ec1cc02b32b4f"
server_key_iv: "dea5e13e6339c4894949ef29231fe576"
server_key_version: "encryptionkey@0"
I20260923 23:14:11.307109 6485 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "73a68c633a4e4b9395b784f7c5bb9920"
format_stamp: "Formatted at 2026-09-23 23:14:11 on dist-test-slave-95hh"
server_key: "f25568f272665f740c0ec1cc02b32b4f"
server_key_iv: "dea5e13e6339c4894949ef29231fe576"
server_key_version: "encryptionkey@0"
I20260923 23:14:11.308368 6485 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:11.309048 6500 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:11.309203 6485 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.001s sys 0.000s
I20260923 23:14:11.309285 6485 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "73a68c633a4e4b9395b784f7c5bb9920"
format_stamp: "Formatted at 2026-09-23 23:14:11 on dist-test-slave-95hh"
server_key: "f25568f272665f740c0ec1cc02b32b4f"
server_key_iv: "dea5e13e6339c4894949ef29231fe576"
server_key_version: "encryptionkey@0"
I20260923 23:14:11.309353 6485 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:11.312672 6485 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:11.314487 6485 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:11.314620 6485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:11.318410 6485 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:34517
I20260923 23:14:11.318452 6552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:34517 every 8 connection(s)
I20260923 23:14:11.318769 6485 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:11.319783 6553 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:11.322079 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920: Bootstrap starting.
I20260923 23:14:11.322674 6553 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:11.322937 6553 log.cc:824] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:11.323216 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6485
I20260923 23:14:11.323307 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:11.323482 16874 external_mini_cluster.cc:1428] Setting key d87f42d8584c755e2624ebe628990165
I20260923 23:14:11.323735 6553 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920: No bootstrap required, opened a new log
I20260923 23:14:11.325736 6553 raft_consensus.cc:348] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } }
I20260923 23:14:11.325881 6553 raft_consensus.cc:374] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260923 23:14:11.325902 6553 raft_consensus.cc:729] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 73a68c633a4e4b9395b784f7c5bb9920, State: Initialized, Role: FOLLOWER
I20260923 23:14:11.326314 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:11.324061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45139 (local address 127.16.122.190:34517)
0923 23:14:11.324256 (+ 195us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:11.324265 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:11.324297 (+ 32us) server_negotiation.cc:407] Connection header received
0923 23:14:11.324953 (+ 656us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:11.324972 (+ 19us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:11.325001 (+ 29us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:11.325130 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:11.325796 (+ 666us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:11.325799 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:11.325800 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:11.325803 (+ 3us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:11.325840 (+ 37us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:11.325851 (+ 11us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:11.325910 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:11.326065 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:14:11.326115 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":137,"thread_start_us":90,"threads_started":1}
I20260923 23:14:11.326031 6553 consensus_queue.cc:260] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } }
I20260923 23:14:11.326377 6553 raft_consensus.cc:388] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260923 23:14:11.326412 6553 raft_consensus.cc:482] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260923 23:14:11.326463 6553 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:11.327188 6553 raft_consensus.cc:504] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } }
I20260923 23:14:11.327253 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:11.327296 6553 leader_election.cc:302] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 73a68c633a4e4b9395b784f7c5bb9920; no voters:
I20260923 23:14:11.327450 6553 leader_election.cc:288] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260923 23:14:11.327495 6558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:11.327639 6558 raft_consensus.cc:686] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [term 1 LEADER]: Becoming Leader. State: Replica: 73a68c633a4e4b9395b784f7c5bb9920, State: Running, Role: LEADER
I20260923 23:14:11.327689 6553 sys_catalog.cc:564] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:11.327764 6558 consensus_queue.cc:237] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } }
I20260923 23:14:11.328091 6558 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 73a68c633a4e4b9395b784f7c5bb9920. Latest consensus state: current_term: 1 leader_uuid: "73a68c633a4e4b9395b784f7c5bb9920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } } }
I20260923 23:14:11.328141 6560 sys_catalog.cc:455] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "73a68c633a4e4b9395b784f7c5bb9920" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "73a68c633a4e4b9395b784f7c5bb9920" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 34517 } } }
I20260923 23:14:11.328223 6558 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:11.328229 6560 sys_catalog.cc:458] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:11.328430 6567 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:11.328946 6567 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:11.330549 6567 catalog_manager.cc:1347] Generated new cluster ID: 6fbefb98c5444e0aaaedaf35f39b4b1b
I20260923 23:14:11.330598 6567 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:11.330874 6557 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:11.329589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51815 (local address 127.16.122.190:34517)
0923 23:14:11.329623 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:11.329625 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:11.329807 (+ 182us) server_negotiation.cc:407] Connection header received
0923 23:14:11.329894 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:11.329894 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:11.329899 (+ 5us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:11.329925 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:11.330579 (+ 654us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:11.330581 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:11.330581 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0923 23:14:11.330585 (+ 4us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:11.330601 (+ 16us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0923 23:14:11.330605 (+ 4us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:11.330655 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:11.330806 (+ 151us) server_negotiation.cc:299] Negotiation successful
0923 23:14:11.330818 (+ 12us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:11.347471 6567 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:11.347669 6567 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:11.351402 6567 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 73a68c633a4e4b9395b784f7c5bb9920: Generated new TSK 0
I20260923 23:14:11.351600 6567 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260923 23:14:11.355197 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6485
2026-09-23T23:14:11Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (1198 ms)
[----------] 18 tests from AuthTokenIssuingTest (21909 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:11 dist-test-slave-95hh krb5kdc[6584](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:11 dist-test-slave-95hh krb5kdc[6584](info): set up 2 sockets
Sep 23 23:14:11 dist-test-slave-95hh krb5kdc[6584](info): commencing operation
krb5kdc: starting...
W20260923 23:14:12.298918 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.926s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:12Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:12.409296 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40835
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:12.490149 6597 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:12.490373 6597 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:12.490398 6597 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:12.490423 6597 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:12.491619 6597 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:12.491659 6597 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:12.491677 6597 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:12.491694 6597 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:12.491708 6597 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:12.493165 6597 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:40835
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6597
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:12.493407 6597 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:12.493610 6597 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:12.498113 6603 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:12.498309 6597 server_base.cc:1034] running on GCE node
W20260923 23:14:12.498126 6605 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:12.498131 6602 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:12.498601 6597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:12.498906 6597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:12.500047 6597 hybrid_clock.cc:648] HybridClock initialized: now 1790205252500025 us; error 36 us; skew 500 ppm
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:12.502452 6597 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:12.502900 6597 webserver.cc:460] Webserver started at http://127.16.122.190:45651/ using document root <none> and password file <none>
I20260923 23:14:12.503122 6597 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:12.503173 6597 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:12.503309 6597 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:12.504359 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "ea75da969aa84e5b9706021cc5b5f070"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "d120f886c6f448c2598fff92afebad4a"
server_key_iv: "883ce8aa0863eb391fd291e53e5dbf9c"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.504685 6597 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "ea75da969aa84e5b9706021cc5b5f070"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "d120f886c6f448c2598fff92afebad4a"
server_key_iv: "883ce8aa0863eb391fd291e53e5dbf9c"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.505905 6597 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:14:12.506667 6612 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:12.506840 6597 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:12.506935 6597 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "ea75da969aa84e5b9706021cc5b5f070"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "d120f886c6f448c2598fff92afebad4a"
server_key_iv: "883ce8aa0863eb391fd291e53e5dbf9c"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.506997 6597 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:12.542655 6597 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:12.544893 6597 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:12.545038 6597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:12.548756 6597 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:40835
I20260923 23:14:12.548790 6664 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:40835 every 8 connection(s)
I20260923 23:14:12.549129 6597 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:12.549962 6665 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:12.551522 6665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:12.554188 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6597
I20260923 23:14:12.554280 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:12.554456 16874 external_mini_cluster.cc:1428] Setting key fb0ad2acecde62e873a5d5b885c18760
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:12.560868 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.552305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40370 (local address 127.16.122.190:40835)
0923 23:14:12.552420 (+ 115us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.552424 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.552451 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:14:12.552923 (+ 472us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.552924 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.552940 (+ 16us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.553014 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.553504 (+ 490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.554338 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.555224 (+ 886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.555539 (+ 315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.555636 (+ 97us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.557723 (+ 2087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.557731 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.557731 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.557739 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.559343 (+ 1604us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.559781 (+ 438us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.559783 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.559784 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.559857 (+ 73us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.560081 (+ 224us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.560081 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.560081 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.560276 (+ 195us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.560341 (+ 65us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.560557 (+ 216us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.560680 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":16,"server-negotiator.queue_time_us":88,"thread_start_us":45,"threads_started":1}
I20260923 23:14:12.560871 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.552158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40835 (local address 127.0.0.1:40370)
0923 23:14:12.552292 (+ 134us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.552328 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.552778 (+ 450us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.553030 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.553032 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.553273 (+ 241us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.553428 (+ 155us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.553432 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.554423 (+ 991us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.554424 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.555153 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.555155 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.555617 (+ 462us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.555618 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.555683 (+ 65us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.556141 (+ 458us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.556149 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.557629 (+ 1480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.559433 (+ 1804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.559441 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.559443 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.559706 (+ 263us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.559912 (+ 206us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.559912 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.559913 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.560021 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.560335 (+ 314us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.560341 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.560430 (+ 89us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.560537 (+ 107us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.560752 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97,"mutex_wait_us":64,"thread_start_us":60,"threads_started":1}
I20260923 23:14:12.561625 6665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.122.189@KRBTEST.COM; defaulting to no policy
I20260923 23:14:12.561977 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.561886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40854)
0923 23:14:12.561917 (+ 31us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.561958 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":16}
W20260923 23:14:12.562098 6615 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
I20260923 23:14:12.562463 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.562381 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40856)
0923 23:14:12.562421 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.562444 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:14:12.562579 6665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36367: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
I20260923 23:14:12.577924 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:36367
--webserver_interface=127.16.122.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:14:12.611790 6665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } attempt: 1
I20260923 23:14:12.612330 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.612194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40862)
0923 23:14:12.612250 (+ 56us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.612295 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:14:12.612725 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.612633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40870)
0923 23:14:12.612678 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.612706 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":30}
W20260923 23:14:12.612831 6665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36367: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
W20260923 23:14:12.657876 6672 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:12.658093 6672 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:12.658119 6672 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:12.658143 6672 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:12.659312 6672 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:12.659351 6672 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:12.659369 6672 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:12.659385 6672 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:12.659400 6672 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:12.660887 6672 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:36367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.122.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6672
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:12.661123 6672 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:12.661336 6672 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:12.666007 6678 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:12.666029 6677 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:12.666007 6680 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:12.666115 6672 server_base.cc:1034] running on GCE node
I20260923 23:14:12.666476 6672 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:12.666740 6672 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:12.667874 6672 hybrid_clock.cc:648] HybridClock initialized: now 1790205252667855 us; error 36 us; skew 500 ppm
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:12.670326 6672 init.cc:377] Logged in from keytab as kudu/127.16.122.189@KRBTEST.COM (short username kudu)
I20260923 23:14:12.670743 6672 webserver.cc:460] Webserver started at http://127.16.122.189:43871/ using document root <none> and password file <none>
I20260923 23:14:12.670953 6672 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:12.670996 6672 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:12.671139 6672 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:12.672210 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data/instance:
uuid: "67794ef383784e2ca7e8befb6d796d94"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "3ea4c9ad38df37401c21c7a326635260"
server_key_iv: "9df3f422857469a46ead85ecc3f291c2"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.672533 6672 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal/instance:
uuid: "67794ef383784e2ca7e8befb6d796d94"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "3ea4c9ad38df37401c21c7a326635260"
server_key_iv: "9df3f422857469a46ead85ecc3f291c2"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.673736 6672 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.002s
I20260923 23:14:12.674360 6687 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:12.674502 6672 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:12.674583 6672 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "67794ef383784e2ca7e8befb6d796d94"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "3ea4c9ad38df37401c21c7a326635260"
server_key_iv: "9df3f422857469a46ead85ecc3f291c2"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.674641 6672 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:12.680977 6665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } attempt: 2
I20260923 23:14:12.681474 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.681337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40882)
0923 23:14:12.681392 (+ 55us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.681437 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":31}
I20260923 23:14:12.681809 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.681729 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40894)
0923 23:14:12.681761 (+ 32us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.681789 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":17}
W20260923 23:14:12.681921 6665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:36367: Network error: Client connection negotiation failed: client connection to 127.16.122.189:36367: connect: Connection refused (error 111)
I20260923 23:14:12.684049 6672 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:12.686051 6672 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:12.686183 6672 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:12.689986 6672 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.189:36367
I20260923 23:14:12.690025 6739 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.189:36367 every 8 connection(s)
I20260923 23:14:12.690357 6672 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
I20260923 23:14:12.691277 6740 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:12.692250 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6672
I20260923 23:14:12.692341 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-1/wal/instance
I20260923 23:14:12.692521 16874 external_mini_cluster.cc:1428] Setting key 148ee38712f51d6a360bed890c49784a
I20260923 23:14:12.692977 6740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.188@KRBTEST.COM" created.
I20260923 23:14:12.701803 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.693723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40382 (local address 127.16.122.190:40835)
0923 23:14:12.693793 (+ 70us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.693796 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.694076 (+ 280us) server_negotiation.cc:407] Connection header received
0923 23:14:12.694748 (+ 672us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.694749 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.694764 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.694816 (+ 52us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.695481 (+ 665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.696018 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.696974 (+ 956us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.697187 (+ 213us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.697242 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.699442 (+ 2200us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.699444 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.699445 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.699453 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.700558 (+ 1105us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.700927 (+ 369us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.700928 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.700928 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.700982 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.701195 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.701196 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.701196 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.701306 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.701361 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.701619 (+ 258us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.701700 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20260923 23:14:12.701964 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.693709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40835 (local address 127.0.0.1:40382)
0923 23:14:12.693930 (+ 221us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.693981 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.694616 (+ 635us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.694894 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.694897 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.695183 (+ 286us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.695394 (+ 211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.695399 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.696110 (+ 711us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.696111 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.696895 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.696897 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.697263 (+ 366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.697264 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.697374 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.697886 (+ 512us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.697894 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.699362 (+ 1468us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.700628 (+ 1266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.700634 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.700636 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.700860 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.701035 (+ 175us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.701036 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.701036 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.701142 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.701360 (+ 218us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.701367 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.701505 (+ 138us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.701594 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.701747 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":176,"thread_start_us":76,"threads_started":1}
I20260923 23:14:12.702513 6740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:12.709610 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.702815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40902)
0923 23:14:12.702857 (+ 42us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.702868 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.702904 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.703164 (+ 260us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.703165 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.703295 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.703340 (+ 45us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.703341 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.704145 (+ 804us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.704146 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.704639 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.704641 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.705129 (+ 488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.705130 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.705192 (+ 62us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.705629 (+ 437us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.705633 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.706934 (+ 1301us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.708603 (+ 1669us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.708603 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.708604 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.708784 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.708978 (+ 194us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.708979 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.708979 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.709032 (+ 53us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.709284 (+ 252us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.709284 (+ 0us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.709321 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.709380 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.709499 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":26}
I20260923 23:14:12.709695 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.702879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40902 (local address 127.16.122.189:36367)
0923 23:14:12.703010 (+ 131us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.703014 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.703060 (+ 46us) server_negotiation.cc:407] Connection header received
0923 23:14:12.703079 (+ 19us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.703081 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.703093 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.703156 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.703400 (+ 244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.704077 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.704703 (+ 626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.705067 (+ 364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.705126 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.707009 (+ 1883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.707017 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.707017 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.707024 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.708531 (+ 1507us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.708871 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.708874 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.708875 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.708926 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.709077 (+ 151us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.709077 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.709078 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.709220 (+ 142us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.709297 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.709452 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.709552 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":25,"server-negotiator.queue_time_us":104,"thread_start_us":82,"threads_started":1}
I20260923 23:14:12.710253 6740 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:12.710647 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.710531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56916)
0923 23:14:12.710574 (+ 43us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.710622 (+ 48us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:14:12.710804 6688 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
I20260923 23:14:12.711221 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.711153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56922)
0923 23:14:12.711181 (+ 28us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.711205 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":15}
W20260923 23:14:12.711323 6740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:36053: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
I20260923 23:14:12.716192 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:36053
--webserver_interface=127.16.122.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45411
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:14:12.760526 6740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } attempt: 1
I20260923 23:14:12.760974 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.760854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56932)
0923 23:14:12.760913 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.760944 (+ 31us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
I20260923 23:14:12.761328 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.761197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56948)
0923 23:14:12.761250 (+ 53us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.761296 (+ 46us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:14:12.761431 6740 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:36053: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
I20260923 23:14:12.773064 6665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } attempt: 3
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:12.779681 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.773459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.16.122.189:36367)
0923 23:14:12.773505 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.773507 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.773518 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:14:12.773576 (+ 58us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.773577 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.773589 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.773632 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.773935 (+ 303us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.774420 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.775121 (+ 701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.775332 (+ 211us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.775384 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.777270 (+ 1886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.777271 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.777272 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.777279 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.778576 (+ 1297us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.779018 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.779019 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.779019 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.779098 (+ 79us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.779269 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.779269 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.779269 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.779367 (+ 98us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.779410 (+ 43us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.779553 (+ 143us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.779614 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:14:12.779740 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.773366 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40910)
0923 23:14:12.773452 (+ 86us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.773464 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.773515 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.773642 (+ 127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.773643 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.773798 (+ 155us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.773861 (+ 63us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.773862 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.774494 (+ 632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.774495 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.775014 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.775016 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.775390 (+ 374us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.775391 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.775471 (+ 80us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.775877 (+ 406us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.775882 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.777178 (+ 1296us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.778660 (+ 1482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.778662 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.778662 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.778944 (+ 282us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.779145 (+ 201us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.779146 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.779146 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.779215 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.779414 (+ 199us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.779415 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.779469 (+ 54us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.779537 (+ 68us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.779645 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":60}
I20260923 23:14:12.780130 6665 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:12.780464 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.780345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56950)
0923 23:14:12.780396 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.780439 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20260923 23:14:12.780781 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.780695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56954)
0923 23:14:12.780733 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.780757 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:14:12.780862 6665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:36053: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
W20260923 23:14:12.798216 6747 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:12.798465 6747 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:12.798482 6747 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:12.798507 6747 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:12.799679 6747 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:12.799719 6747 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:12.799732 6747 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:12.799743 6747 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:12.799759 6747 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:12.801211 6747 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45411
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:40835,127.16.122.189:36367,127.16.122.188:36053
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:36053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.122.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6747
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:12.801411 6747 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:12.801584 6747 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:12.806545 6752 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:12.806638 6753 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:12.806610 6747 server_base.cc:1034] running on GCE node
W20260923 23:14:12.806553 6755 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:12.807013 6747 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:12.807346 6747 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:12.808518 6747 hybrid_clock.cc:648] HybridClock initialized: now 1790205252808504 us; error 30 us; skew 500 ppm
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:12.811344 6747 init.cc:377] Logged in from keytab as kudu/127.16.122.188@KRBTEST.COM (short username kudu)
I20260923 23:14:12.811836 6747 webserver.cc:460] Webserver started at http://127.16.122.188:38853/ using document root <none> and password file <none>
I20260923 23:14:12.811985 6665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } attempt: 1
I20260923 23:14:12.812055 6747 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:12.812105 6747 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:12.812240 6747 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:12.812371 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.812220 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56958)
0923 23:14:12.812290 (+ 70us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.812333 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20260923 23:14:12.812711 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.812607 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56974)
0923 23:14:12.812652 (+ 45us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.812684 (+ 32us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:14:12.812816 6665 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:36053: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
I20260923 23:14:12.813623 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data/instance:
uuid: "9fb29a7d8b624e7caba972785ca1f7b1"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "8a721d3f79cb5d0e1a5676cd92c66a59"
server_key_iv: "c8d2897cd75b251f2715e25734c6b280"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.813957 6747 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal/instance:
uuid: "9fb29a7d8b624e7caba972785ca1f7b1"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "8a721d3f79cb5d0e1a5676cd92c66a59"
server_key_iv: "c8d2897cd75b251f2715e25734c6b280"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.815541 6747 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:14:12.816339 6762 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:12.816488 6747 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:12.816576 6747 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "9fb29a7d8b624e7caba972785ca1f7b1"
format_stamp: "Formatted at 2026-09-23 23:14:12 on dist-test-slave-95hh"
server_key: "8a721d3f79cb5d0e1a5676cd92c66a59"
server_key_iv: "c8d2897cd75b251f2715e25734c6b280"
server_key_version: "encryptionkey@0"
I20260923 23:14:12.816648 6747 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:12.823582 6747 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:12.825850 6747 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:12.825994 6747 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:12.829577 6740 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } attempt: 2
I20260923 23:14:12.830078 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.829928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:56990)
0923 23:14:12.829995 (+ 67us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.830040 (+ 45us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:36053: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20260923 23:14:12.830416 6747 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.188:36053
I20260923 23:14:12.830433 6814 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.188:36053 every 8 connection(s)
I20260923 23:14:12.830881 6747 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
I20260923 23:14:12.831586 6815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:12.833204 6815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:12.839396 6743 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.830460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:57000)
0923 23:14:12.830511 (+ 51us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.830522 (+ 11us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.830576 (+ 54us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.832132 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.832134 (+ 2us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.832297 (+ 163us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.832350 (+ 53us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.832352 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.833440 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.833441 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.834157 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.834161 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.834754 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.834755 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.834801 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.835248 (+ 447us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.835253 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.836601 (+ 1348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.838306 (+ 1705us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.838307 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.838307 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.838459 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.838686 (+ 227us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.838686 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.838687 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.838744 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.839092 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.839093 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.839138 (+ 45us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.839201 (+ 63us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.839299 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 23:14:12.839618 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.830793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57000 (local address 127.16.122.188:36053)
0923 23:14:12.831311 (+ 518us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.831319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.831347 (+ 28us) server_negotiation.cc:407] Connection header received
0923 23:14:12.831919 (+ 572us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.831927 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.831993 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.832114 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.832453 (+ 339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.833353 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.834227 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.834674 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.834789 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.836682 (+ 1893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.836688 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.836689 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.836698 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.838223 (+ 1525us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.838544 (+ 321us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.838546 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.838547 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.838622 (+ 75us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.838793 (+ 171us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.838794 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.838795 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.839015 (+ 220us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.839110 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.839265 (+ 155us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.839398 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":455,"threads_started":1}
I20260923 23:14:12.840781 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.833884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40398 (local address 127.16.122.190:40835)
0923 23:14:12.833933 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.833935 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.834061 (+ 126us) server_negotiation.cc:407] Connection header received
0923 23:14:12.834123 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.834124 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.834132 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.834174 (+ 42us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.834595 (+ 421us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.835270 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.836184 (+ 914us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.836452 (+ 268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.836504 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.838418 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.838420 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.838420 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.838427 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.839698 (+ 1271us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.840033 (+ 335us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.840035 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.840035 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.840098 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.840276 (+ 178us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.840277 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.840278 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.840391 (+ 113us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.840446 (+ 55us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.840629 (+ 183us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.840699 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":28}
I20260923 23:14:12.840854 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6747
I20260923 23:14:12.840879 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.833788 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:40835 (local address 127.0.0.1:40398)
0923 23:14:12.833956 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.833998 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.834067 (+ 69us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.834193 (+ 126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.834196 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.834458 (+ 262us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.834527 (+ 69us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.834528 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.835350 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.835351 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.836081 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.836084 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.836531 (+ 447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.836532 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.836618 (+ 86us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.837038 (+ 420us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.837042 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.838349 (+ 1307us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.839774 (+ 1425us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.839778 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.839780 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.839969 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.840159 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.840160 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.840160 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.840217 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.840444 (+ 227us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.840451 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.840497 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.840587 (+ 90us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.840694 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":140,"thread_start_us":104,"threads_started":1}
I20260923 23:14:12.840948 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1790204999454807-16874-0/minicluster-data/master-2/wal/instance
I20260923 23:14:12.841146 16874 external_mini_cluster.cc:1428] Setting key a058371553e17724307c5ce7b8ec4073
I20260923 23:14:12.841481 6740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: Bootstrap starting.
I20260923 23:14:12.842625 6740 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:12.842891 6740 log.cc:824] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:12.843458 6815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:12.843921 6740 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: No bootstrap required, opened a new log
I20260923 23:14:12.845901 6740 raft_consensus.cc:348] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:14:12.846101 6740 raft_consensus.cc:729] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 67794ef383784e2ca7e8befb6d796d94, State: Initialized, Role: FOLLOWER
I20260923 23:14:12.846205 6740 consensus_queue.cc:260] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.846555 6740 sys_catalog.cc:564] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:12.846544 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.846724 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: This master's current role is: FOLLOWER
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
W20260923 23:14:12.848954 6832 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:12.849050 6832 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:12.849606 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.841488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57006 (local address 127.16.122.188:36053)
0923 23:14:12.841525 (+ 37us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.841527 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.841716 (+ 189us) server_negotiation.cc:407] Connection header received
0923 23:14:12.841792 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.841792 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.841802 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.841845 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.842551 (+ 706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.842984 (+ 433us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.843903 (+ 919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.844174 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.844241 (+ 67us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.846580 (+ 2339us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.846582 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.846582 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.846589 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.848520 (+ 1931us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.848904 (+ 384us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.848905 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.848906 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.848970 (+ 64us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.849157 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.849158 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.849159 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.849260 (+ 101us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.849408 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.849454 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.849526 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:12.850386 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.843696 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:36367 (local address 127.0.0.1:40920)
0923 23:14:12.843773 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.843782 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.843823 (+ 41us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.843925 (+ 102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.843926 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.844063 (+ 137us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.844109 (+ 46us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.844110 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.844864 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.844865 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.845583 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.845586 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.846037 (+ 451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.846038 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.846105 (+ 67us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.846478 (+ 373us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.846483 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.847730 (+ 1247us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.849394 (+ 1664us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.849395 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.849395 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.849584 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.849774 (+ 190us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.849775 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.849775 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.849826 (+ 51us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.850103 (+ 277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.850104 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.850146 (+ 42us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.850206 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.850302 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":63}
I20260923 23:14:12.850899 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.843737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40920 (local address 127.16.122.189:36367)
0923 23:14:12.843773 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.843775 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.843811 (+ 36us) server_negotiation.cc:407] Connection header received
0923 23:14:12.843863 (+ 52us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.843864 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.843873 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.843922 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.844172 (+ 250us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.844776 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.845678 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.845960 (+ 282us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.846025 (+ 65us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.847855 (+ 1830us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.847857 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.847857 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.847864 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.849308 (+ 1444us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.849650 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.849651 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.849652 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.849703 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.849888 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.849889 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.849889 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.850037 (+ 148us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.850097 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.850747 (+ 650us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.850819 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:14:12.848793 6672 master_runner.cc:186] Error getting master registration for 127.16.122.190:40835: OK, instance_id { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" instance_seqno: 1790205252547676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:14:12.851372 6672 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:14:12.851580 6815 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:14:12.857828 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.850237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40408 (local address 127.16.122.190:40835)
0923 23:14:12.850277 (+ 40us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.850279 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.850286 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:14:12.850307 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.850308 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.850316 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.850365 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.850892 (+ 527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.851567 (+ 675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.852390 (+ 823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.852661 (+ 271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.852722 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.855108 (+ 2386us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.855110 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.855110 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.855117 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.856619 (+ 1502us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.856978 (+ 359us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.856979 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.856980 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.857038 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.857225 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.857226 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.857226 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.857353 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.857535 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.857571 (+ 36us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.857638 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21}
I20260923 23:14:12.858458 6818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.851803 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:57014)
0923 23:14:12.851860 (+ 57us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.851868 (+ 8us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.851904 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.852009 (+ 105us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.852010 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.852136 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.852165 (+ 29us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.852167 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.852777 (+ 610us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.852778 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.853750 (+ 972us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.853753 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.854158 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.854159 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.854205 (+ 46us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.854513 (+ 308us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.854517 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.855967 (+ 1450us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.857531 (+ 1564us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.857532 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.857532 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.857703 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.857866 (+ 163us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.857867 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.857867 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.857912 (+ 45us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.858184 (+ 272us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.858185 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.858218 (+ 33us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.858272 (+ 54us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.858369 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20260923 23:14:12.858702 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.851831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57014 (local address 127.16.122.188:36053)
0923 23:14:12.851865 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.851867 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.851894 (+ 27us) server_negotiation.cc:407] Connection header received
0923 23:14:12.851945 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.851946 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.851953 (+ 7us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.852007 (+ 54us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.852222 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.852707 (+ 485us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.853854 (+ 1147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.854097 (+ 243us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.854151 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.856038 (+ 1887us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.856039 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.856040 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.856046 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.857462 (+ 1416us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.857770 (+ 308us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.857771 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.857771 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.857814 (+ 43us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.857964 (+ 150us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.857964 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.857965 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.858112 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.858165 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.858560 (+ 395us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.858628 (+ 68us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15}
I20260923 23:14:12.861034 6815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: Bootstrap starting.
I20260923 23:14:12.862118 6815 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:12.862437 6815 log.cc:824] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: Log is configured to *not* fsync() on all Append() calls
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:12.863291 6815 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: No bootstrap required, opened a new log
I20260923 23:14:12.865038 6815 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.865199 6815 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9fb29a7d8b624e7caba972785ca1f7b1, State: Initialized, Role: FOLLOWER
I20260923 23:14:12.865336 6815 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.865711 6815 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:12.866056 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.858329 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40926 (local address 127.16.122.189:36367)
0923 23:14:12.858367 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.858369 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.858376 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:14:12.858399 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.858399 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.858408 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.858464 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.858977 (+ 513us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.859729 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.860530 (+ 801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.860813 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.860879 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.863433 (+ 2554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.863435 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.863435 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.863442 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.864922 (+ 1480us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.865300 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.865302 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.865302 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.865359 (+ 57us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.865561 (+ 202us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.865562 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.865562 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.865695 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.865845 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.865889 (+ 44us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.865970 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:14:12.867759 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:12.868966 6639 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "ea75da969aa84e5b9706021cc5b5f070"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40408
I20260923 23:14:12.867717 6747 master_runner.cc:186] Error getting master registration for 127.16.122.190:40835: OK, instance_id { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" instance_seqno: 1790205252547676 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:14:12.870431 6747 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:14:12.871186 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.871292 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:14:12.871950 6846 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:12.871999 6846 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:12.887964 6665 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } attempt: 2
Sep 23 23:14:12 dist-test-slave-95hh krb5kdc[6584](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205252, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:14:12.896275 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.888483 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57016 (local address 127.16.122.188:36053)
0923 23:14:12.888527 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.888529 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.888539 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:14:12.888568 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.888569 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.888580 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.888656 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.888996 (+ 340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.889847 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.890850 (+ 1003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.891188 (+ 338us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.891249 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.893489 (+ 2240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.893491 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.893491 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.893499 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.895115 (+ 1616us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.895514 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.895515 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.895516 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.895576 (+ 60us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.895761 (+ 185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.895762 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.895762 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.895895 (+ 133us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.895957 (+ 62us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.896095 (+ 138us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.896180 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:14:12.896275 6667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.888391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:36053 (local address 127.0.0.1:57016)
0923 23:14:12.888450 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:12.888462 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:12.888501 (+ 39us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:12.888673 (+ 172us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:12.888673 (+ 0us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:12.888875 (+ 202us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:12.888931 (+ 56us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.888933 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.889941 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.889942 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.890759 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:12.890762 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.891274 (+ 512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.891275 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:12.891341 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.891835 (+ 494us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:12.891840 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:12.893381 (+ 1541us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:12.895196 (+ 1815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.895197 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.895198 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.895441 (+ 243us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.895629 (+ 188us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:12.895630 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:12.895630 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:12.895703 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:12.895956 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:12.895957 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:12.896004 (+ 47us) client_negotiation.cc:769] Sending connection context
0923 23:14:12.896068 (+ 64us) client_negotiation.cc:240] Negotiation successful
0923 23:14:12.896172 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:14:12.898458 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070: Bootstrap starting.
I20260923 23:14:12.899247 6665 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:12.899544 6665 log.cc:824] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:12.900426 6665 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070: No bootstrap required, opened a new log
I20260923 23:14:12.901841 6665 raft_consensus.cc:348] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.902074 6665 raft_consensus.cc:729] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ea75da969aa84e5b9706021cc5b5f070, State: Initialized, Role: FOLLOWER
I20260923 23:14:12.902232 6665 consensus_queue.cc:260] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.902680 6665 sys_catalog.cc:564] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:12.902642 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.902779 6639 raft_consensus.cc:482] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 23:14:12.902871 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:12.903003 6639 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:12.903893 6639 raft_consensus.cc:504] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.905804 6714 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea75da969aa84e5b9706021cc5b5f070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "67794ef383784e2ca7e8befb6d796d94"
I20260923 23:14:12.905829 6789 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ea75da969aa84e5b9706021cc5b5f070" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "9fb29a7d8b624e7caba972785ca1f7b1"
I20260923 23:14:12.905929 6714 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:12.905953 6789 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:12.906796 6714 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea75da969aa84e5b9706021cc5b5f070 in term 1.
I20260923 23:14:12.907078 6615 leader_election.cc:302] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 67794ef383784e2ca7e8befb6d796d94, ea75da969aa84e5b9706021cc5b5f070; no voters:
I20260923 23:14:12.907222 6849 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:12.907310 6789 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ea75da969aa84e5b9706021cc5b5f070 in term 1.
I20260923 23:14:12.907362 6849 raft_consensus.cc:686] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [term 1 LEADER]: Becoming Leader. State: Replica: ea75da969aa84e5b9706021cc5b5f070, State: Running, Role: LEADER
I20260923 23:14:12.907445 6849 consensus_queue.cc:237] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } }
I20260923 23:14:12.907725 6639 leader_election.cc:288] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [CANDIDATE]: Term 1 election: Requested vote from peers 67794ef383784e2ca7e8befb6d796d94 (127.16.122.189:36367), 9fb29a7d8b624e7caba972785ca1f7b1 (127.16.122.188:36053)
I20260923 23:14:12.907886 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea75da969aa84e5b9706021cc5b5f070. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.907959 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:12.908140 6862 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:12.908607 6862 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:12.910746 6789 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [term 1 FOLLOWER]: Refusing update from remote peer ea75da969aa84e5b9706021cc5b5f070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:14:12.911082 6849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:14:12.912523 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea75da969aa84e5b9706021cc5b5f070. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.912606 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:12.913659 6714 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [term 1 FOLLOWER]: Refusing update from remote peer ea75da969aa84e5b9706021cc5b5f070: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:14:12.913921 6849 consensus_queue.cc:1035] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [LEADER]: Connected to new peer: Peer: permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:14:12.915074 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ea75da969aa84e5b9706021cc5b5f070. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.915158 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:12.917924 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.918015 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:12.918536 6835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.918622 6835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:12.918887 6870 mvcc.cc:205] Tried to move back new op lower bound from 7332680715920031744 to 7332680715909664768. Current Snapshot: MvccSnapshot[applied={T|T < 7332680715920031744}]
I20260923 23:14:12.919123 6869 mvcc.cc:205] Tried to move back new op lower bound from 7332680715920031744 to 7332680715909664768. Current Snapshot: MvccSnapshot[applied={T|T < 7332680715920031744}]
I20260923 23:14:12.920212 6849 sys_catalog.cc:455] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.920302 6849 sys_catalog.cc:458] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070 [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:12.920715 6821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ea75da969aa84e5b9706021cc5b5f070" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ea75da969aa84e5b9706021cc5b5f070" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 40835 } } peers { permanent_uuid: "67794ef383784e2ca7e8befb6d796d94" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 36367 } } peers { permanent_uuid: "9fb29a7d8b624e7caba972785ca1f7b1" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 36053 } } }
I20260923 23:14:12.920817 6821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:12.920882 6875 mvcc.cc:205] Tried to move back new op lower bound from 7332680715920031744 to 7332680715909664768. Current Snapshot: MvccSnapshot[applied={T|T < 7332680715920031744}]
I20260923 23:14:12.923381 6668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.912971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40422 (local address 127.16.122.190:40835)
0923 23:14:12.913010 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.913012 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.913469 (+ 457us) server_negotiation.cc:407] Connection header received
0923 23:14:12.913566 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.913566 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.913577 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.913604 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.914494 (+ 890us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.915275 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.916390 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.916688 (+ 298us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.916726 (+ 38us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.920319 (+ 3593us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.920320 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.920320 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.920328 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.922116 (+ 1788us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.922537 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.922538 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.922538 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.922592 (+ 54us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.922778 (+ 186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.922779 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.922779 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.922904 (+ 125us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.922984 (+ 80us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.923210 (+ 226us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.923300 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19}
I20260923 23:14:12.924294 6862 catalog_manager.cc:1347] Generated new cluster ID: c327a97f72a943199b0b71e809c15067
I20260923 23:14:12.924351 6862 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260923 23:14:12.941043 6745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.913196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40932 (local address 127.16.122.189:36367)
0923 23:14:12.913230 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.913232 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.915798 (+ 2566us) server_negotiation.cc:407] Connection header received
0923 23:14:12.916126 (+ 328us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.916127 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.916136 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.916161 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.919144 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.919799 (+ 655us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.923023 (+ 3224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.923336 (+ 313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.923382 (+ 46us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.935158 (+ 11776us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.935160 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.935161 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.935170 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.936920 (+ 1750us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.939469 (+ 2549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.939470 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.939471 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.939537 (+ 66us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.939928 (+ 391us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.939929 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.939929 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.940080 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.940180 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.940849 (+ 669us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.940946 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:14:12.942279 6817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:12.909883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57030 (local address 127.16.122.188:36053)
0923 23:14:12.909921 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:12.909923 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:12.915894 (+ 5971us) server_negotiation.cc:407] Connection header received
0923 23:14:12.916029 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:12.916030 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:12.916040 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:12.916065 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:12.919899 (+ 3834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.920533 (+ 634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.922605 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:12.922897 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:12.922942 (+ 45us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:12.937383 (+ 14441us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:12.937385 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:12.937385 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:12.937400 (+ 15us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:12.939006 (+ 1606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.939779 (+ 773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.939780 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.939780 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.939842 (+ 62us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:12.940512 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:12.940514 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:12.940514 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:12.940654 (+ 140us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:12.940763 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:12.942090 (+ 1327us) server_negotiation.cc:299] Negotiation successful
0923 23:14:12.942177 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17}
I20260923 23:14:12.960151 6862 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:12.960913 6862 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:12.971930 6862 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ea75da969aa84e5b9706021cc5b5f070: Generated new TSK 0
I20260923 23:14:12.972163 6862 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:14:13.028977 6668 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.027711 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40432 (local address 127.16.122.190:40835)
0923 23:14:13.027750 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.027752 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.028376 (+ 624us) server_negotiation.cc:407] Connection header received
0923 23:14:13.028403 (+ 27us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.028404 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.028413 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.028461 (+ 48us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.028901 (+ 440us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40432: BlockingRecv error: recv got EOF from 127.0.0.1:40432 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:14:13.028986 6817 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.027944 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57046 (local address 127.16.122.188:36053)
0923 23:14:13.027986 (+ 42us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.027987 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.028417 (+ 430us) server_negotiation.cc:407] Connection header received
0923 23:14:13.028506 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.028506 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.028517 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.028544 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.028924 (+ 380us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57046: BlockingRecv error: recv got EOF from 127.0.0.1:57046 (error 108)
Metrics: {"server-negotiator.queue_time_us":24}
W20260923 23:14:13.029774 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.027399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40948 (local address 127.16.122.189:36367)
0923 23:14:13.027446 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.027448 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.028164 (+ 716us) server_negotiation.cc:407] Connection header received
0923 23:14:13.028233 (+ 69us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.028234 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.028247 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.028336 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.029706 (+ 1370us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40948: BlockingRecv error: recv got EOF from 127.0.0.1:40948 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:14:13.032160 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.031432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40958 (local address 127.16.122.189:36367)
0923 23:14:13.031467 (+ 35us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.031468 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.031692 (+ 224us) server_negotiation.cc:407] Connection header received
0923 23:14:13.031792 (+ 100us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.031793 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.031811 (+ 18us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.031839 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.032113 (+ 274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40958: BlockingRecv error: recv got EOF from 127.0.0.1:40958 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 23:14:13.035342 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.034518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40962 (local address 127.16.122.189:36367)
0923 23:14:13.034554 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.034555 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.034800 (+ 245us) server_negotiation.cc:407] Connection header received
0923 23:14:13.034895 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.034896 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.034907 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.034932 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.035292 (+ 360us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40962: BlockingRecv error: recv got EOF from 127.0.0.1:40962 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 23:14:13.039348 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.038564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40968 (local address 127.16.122.189:36367)
0923 23:14:13.038598 (+ 34us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.038599 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.038831 (+ 232us) server_negotiation.cc:407] Connection header received
0923 23:14:13.038920 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.038921 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.038931 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.038956 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.039298 (+ 342us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40968: BlockingRecv error: recv got EOF from 127.0.0.1:40968 (error 108)
Metrics: {"server-negotiator.queue_time_us":17}
W20260923 23:14:13.045225 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.044500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40972 (local address 127.16.122.189:36367)
0923 23:14:13.044536 (+ 36us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.044538 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.044757 (+ 219us) server_negotiation.cc:407] Connection header received
0923 23:14:13.044841 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.044842 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.044853 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.044880 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.045174 (+ 294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40972: BlockingRecv error: recv got EOF from 127.0.0.1:40972 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 23:14:13.055425 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.054482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40988 (local address 127.16.122.189:36367)
0923 23:14:13.054527 (+ 45us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.054528 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.054790 (+ 262us) server_negotiation.cc:407] Connection header received
0923 23:14:13.054899 (+ 109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.054901 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.054914 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.054942 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.055368 (+ 426us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40988: BlockingRecv error: recv got EOF from 127.0.0.1:40988 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 23:14:13.073354 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.072685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40992 (local address 127.16.122.189:36367)
0923 23:14:13.072735 (+ 50us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.072736 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.072868 (+ 132us) server_negotiation.cc:407] Connection header received
0923 23:14:13.072925 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.072927 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.072939 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.072985 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.073290 (+ 305us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40992: BlockingRecv error: recv got EOF from 127.0.0.1:40992 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:14:13.107009 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.106367 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41002 (local address 127.16.122.189:36367)
0923 23:14:13.106411 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.106413 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.106534 (+ 121us) server_negotiation.cc:407] Connection header received
0923 23:14:13.106594 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.106595 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.106607 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.106652 (+ 45us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.106951 (+ 299us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41002: BlockingRecv error: recv got EOF from 127.0.0.1:41002 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 23:14:13.172673 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.172026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41008 (local address 127.16.122.189:36367)
0923 23:14:13.172098 (+ 72us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.172101 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.172154 (+ 53us) server_negotiation.cc:407] Connection header received
0923 23:14:13.172240 (+ 86us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.172241 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.172253 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.172296 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.172611 (+ 315us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41008: BlockingRecv error: recv got EOF from 127.0.0.1:41008 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
W20260923 23:14:13.302457 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.301753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41014 (local address 127.16.122.189:36367)
0923 23:14:13.301831 (+ 78us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.301833 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.301882 (+ 49us) server_negotiation.cc:407] Connection header received
0923 23:14:13.301932 (+ 50us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.301933 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.301944 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.301994 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.302377 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41014: BlockingRecv error: recv got EOF from 127.0.0.1:41014 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20260923 23:14:13.560177 6745 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:13.559611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41020 (local address 127.16.122.189:36367)
0923 23:14:13.559682 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:13.559684 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:13.559694 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:14:13.559756 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:13.559757 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:13.559768 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:13.559817 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:13.560112 (+ 295us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41020: BlockingRecv error: recv got EOF from 127.0.0.1:41020 (error 108)
Metrics: {"server-negotiator.queue_time_us":42}
I20260923 23:14:13.849525 6832 catalog_manager.cc:1259] Loaded cluster ID: c327a97f72a943199b0b71e809c15067
I20260923 23:14:13.849601 6832 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: loading cluster ID for follower catalog manager: success
I20260923 23:14:13.850373 6832 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: acquiring CA information for follower catalog manager: success
I20260923 23:14:13.850616 6832 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 67794ef383784e2ca7e8befb6d796d94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:14:13.872423 6846 catalog_manager.cc:1259] Loaded cluster ID: c327a97f72a943199b0b71e809c15067
I20260923 23:14:13.872496 6846 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: loading cluster ID for follower catalog manager: success
I20260923 23:14:13.873297 6846 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: acquiring CA information for follower catalog manager: success
I20260923 23:14:13.873587 6846 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 9fb29a7d8b624e7caba972785ca1f7b1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:14:14.075688 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:14.073268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41032 (local address 127.16.122.189:36367)
0923 23:14:14.073382 (+ 114us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:14.073384 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:14.073410 (+ 26us) server_negotiation.cc:407] Connection header received
0923 23:14:14.073482 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:14.073483 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:14.073535 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:14.073604 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:14.073714 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.074254 (+ 540us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.074982 (+ 728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.075218 (+ 236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.075267 (+ 49us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:14.075411 (+ 144us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:14.075520 (+ 109us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:14.075574 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:14.075587 (+ 13us) server_negotiation.cc:299] Negotiation successful
0923 23:14:14.075605 (+ 18us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92,"thread_start_us":56,"threads_started":1}
I20260923 23:14:14.079794 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:14.077500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57054 (local address 127.16.122.188:36053)
0923 23:14:14.077603 (+ 103us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:14.077605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:14.077613 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:14:14.077711 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:14.077712 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:14.077746 (+ 34us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:14.077805 (+ 59us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:14.077877 (+ 72us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.078399 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.079116 (+ 717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.079318 (+ 202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.079365 (+ 47us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:14.079491 (+ 126us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:14.079592 (+ 101us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:14.079642 (+ 50us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:14.079699 (+ 57us) server_negotiation.cc:299] Negotiation successful
0923 23:14:14.079714 (+ 15us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83,"thread_start_us":47,"threads_started":1}
I20260923 23:14:14.083750 6945 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:14.081224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41042 (local address 127.16.122.189:36367)
0923 23:14:14.081265 (+ 41us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:14.081267 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:14.081408 (+ 141us) server_negotiation.cc:407] Connection header received
0923 23:14:14.081493 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:14.081494 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:14.081505 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:14.081554 (+ 49us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:14.081677 (+ 123us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.082402 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.083115 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.083324 (+ 209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.083375 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:14.083508 (+ 133us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:14.083542 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:14.083584 (+ 42us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:14.083646 (+ 62us) server_negotiation.cc:299] Negotiation successful
0923 23:14:14.083674 (+ 28us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18}
I20260923 23:14:14.084229 6957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:14.081243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40442 (local address 127.16.122.190:40835)
0923 23:14:14.081442 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:14.081444 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:14.081614 (+ 170us) server_negotiation.cc:407] Connection header received
0923 23:14:14.081663 (+ 49us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:14.081664 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:14.081712 (+ 48us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:14.081783 (+ 71us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:14.081893 (+ 110us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.082363 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.083116 (+ 753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.083411 (+ 295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.083477 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:14.083627 (+ 150us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:14.083804 (+ 177us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:14.083861 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:14.084123 (+ 262us) server_negotiation.cc:299] Negotiation successful
0923 23:14:14.084146 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181,"thread_start_us":50,"threads_started":1}
I20260923 23:14:14.085161 6951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:14.081380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57056 (local address 127.16.122.188:36053)
0923 23:14:14.081410 (+ 30us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:14.081412 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:14.081418 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:14:14.081439 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:14.081440 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:14.081449 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:14.081493 (+ 44us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:14.081622 (+ 129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.082350 (+ 728us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.083947 (+ 1597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:14.084281 (+ 334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:14.084342 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:14.084676 (+ 334us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:14.084706 (+ 30us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:14.084760 (+ 54us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:14.085060 (+ 300us) server_negotiation.cc:299] Negotiation successful
0923 23:14:14.085085 (+ 25us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16}
I20260923 23:14:14.085961 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6597
I20260923 23:14:14.090504 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6672
I20260923 23:14:14.094997 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6747
2026-09-23T23:14:14Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (2740 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 23 23:14:14 dist-test-slave-95hh krb5kdc[6962](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 23 23:14:14 dist-test-slave-95hh krb5kdc[6962](info): set up 2 sockets
Sep 23 23:14:14 dist-test-slave-95hh krb5kdc[6962](info): commencing operation
krb5kdc: starting...
W20260923 23:14:15.039111 16874 mini_kdc.cc:121] Time spent starting KDC: real 0.925s user 0.000s sys 0.005s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/test-user.keytab.
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-23T23:14:15Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-09-23T23:14:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.122.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
Entry for principal kudu/127.16.122.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab.
I20260923 23:14:15.152022 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36731
--webserver_interface=127.16.122.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260923 23:14:15.231889 6975 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:15.232117 6975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:15.232136 6975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:15.232155 6975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:15.233284 6975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:15.233318 6975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:15.233331 6975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:15.233341 6975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:15.233352 6975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:15.234732 6975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.190.keytab
--principal=kudu/127.16.122.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.190:36731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.122.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.6975
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:15.234918 6975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:15.235142 6975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:14:15.239789 6975 server_base.cc:1034] running on GCE node
W20260923 23:14:15.239761 6983 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:15.239732 6980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:15.239754 6981 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:15.240247 6975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:15.240502 6975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:15.241655 6975 hybrid_clock.cc:648] HybridClock initialized: now 1790205255241634 us; error 36 us; skew 500 ppm
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:15.244027 6975 init.cc:377] Logged in from keytab as kudu/127.16.122.190@KRBTEST.COM (short username kudu)
I20260923 23:14:15.244609 6975 webserver.cc:460] Webserver started at http://127.16.122.190:37089/ using document root <none> and password file <none>
I20260923 23:14:15.244817 6975 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:15.244867 6975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:15.245002 6975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:15.246133 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data/instance:
uuid: "2f52918c93c94a87a4394306f3c46c8a"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1b66a300dda2160bc818fdefce05101c"
server_key_iv: "82474cb4ecebbc5dc0a8add9d8285e9a"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.246492 6975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance:
uuid: "2f52918c93c94a87a4394306f3c46c8a"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1b66a300dda2160bc818fdefce05101c"
server_key_iv: "82474cb4ecebbc5dc0a8add9d8285e9a"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.247880 6975 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.003s
I20260923 23:14:15.248571 6990 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:15.248724 6975 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:15.248805 6975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal
uuid: "2f52918c93c94a87a4394306f3c46c8a"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1b66a300dda2160bc818fdefce05101c"
server_key_iv: "82474cb4ecebbc5dc0a8add9d8285e9a"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.248867 6975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:15.268262 6975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:15.270113 6975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:15.270260 6975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:15.274770 6975 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.190:36731
I20260923 23:14:15.274818 7042 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.190:36731 every 8 connection(s)
I20260923 23:14:15.275313 6975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/data/info.pb
I20260923 23:14:15.276240 7043 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:15.276675 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 6975
I20260923 23:14:15.276757 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-0/wal/instance
I20260923 23:14:15.276938 16874 external_mini_cluster.cc:1428] Setting key 314c892af7883c21e232d7c5e42f3a36
I20260923 23:14:15.277962 7043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.122.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.189@KRBTEST.COM" created.
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:15.290668 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.278653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36731 (local address 127.0.0.1:42434)
0923 23:14:15.278812 (+ 159us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.278855 (+ 43us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.279386 (+ 531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.279684 (+ 298us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.279687 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.279933 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.280093 (+ 160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.280098 (+ 5us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.281591 (+ 1493us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.281592 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.282280 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.282282 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.282929 (+ 647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.282929 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.282980 (+ 51us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.283561 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.283569 (+ 8us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.287376 (+ 3807us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.289260 (+ 1884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.289266 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.289268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.289457 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.289698 (+ 241us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.289699 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.289699 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.289773 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.290173 (+ 400us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.290177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.290246 (+ 69us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.290322 (+ 76us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.290451 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":116,"thread_start_us":74,"threads_started":1}
I20260923 23:14:15.290676 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.278845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42434 (local address 127.16.122.190:36731)
0923 23:14:15.278986 (+ 141us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.278990 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.279054 (+ 64us) server_negotiation.cc:407] Connection header received
0923 23:14:15.279538 (+ 484us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.279540 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.279570 (+ 30us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.279685 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.280185 (+ 500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.281480 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.282371 (+ 891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.282854 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.282967 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.287460 (+ 4493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.287468 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.287468 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.287475 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.289178 (+ 1703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.289526 (+ 348us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.289529 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.289530 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.289642 (+ 112us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.289824 (+ 182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.289825 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.289825 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.290110 (+ 285us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.290197 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.290374 (+ 177us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.290533 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":55,"server-negotiator.queue_time_us":106,"thread_start_us":58,"threads_started":1}
I20260923 23:14:15.291422 7043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:15.291771 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.291673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58602)
0923 23:14:15.291713 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.291753 (+ 40us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:14:15.291906 6992 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
I20260923 23:14:15.292303 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.292217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58606)
0923 23:14:15.292264 (+ 47us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.292288 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:14:15.292469 7043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:42301: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
Entry for principal kudu/127.16.122.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab.
I20260923 23:14:15.301391 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:42301
--webserver_interface=127.16.122.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:14:15.341688 7043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } attempt: 1
I20260923 23:14:15.342270 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.342135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58610)
0923 23:14:15.342197 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.342240 (+ 43us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20260923 23:14:15.342630 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.342550 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58624)
0923 23:14:15.342588 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.342612 (+ 24us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:14:15.342736 7043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:42301: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
W20260923 23:14:15.381058 7050 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:15.381285 7050 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:15.381301 7050 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:15.381321 7050 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:15.382463 7050 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:15.382501 7050 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:15.382513 7050 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:15.382524 7050 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:15.382539 7050 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:15.384018 7050 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.189.keytab
--principal=kudu/127.16.122.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.189:42301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.122.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7050
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:15.384227 7050 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:15.384404 7050 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260923 23:14:15.388985 7055 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:15.388985 7058 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:15.389001 7050 server_base.cc:1034] running on GCE node
W20260923 23:14:15.389084 7056 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:15.389427 7050 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:15.389727 7050 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:15.390868 7050 hybrid_clock.cc:648] HybridClock initialized: now 1790205255390857 us; error 28 us; skew 500 ppm
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:15.393285 7050 init.cc:377] Logged in from keytab as kudu/127.16.122.189@KRBTEST.COM (short username kudu)
I20260923 23:14:15.393684 7050 webserver.cc:460] Webserver started at http://127.16.122.189:32951/ using document root <none> and password file <none>
I20260923 23:14:15.393893 7050 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:15.393939 7050 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:15.394059 7050 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:15.395148 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data/instance:
uuid: "2782cbf53f7f4b269547f7f02b84c785"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1bb0868c0c59fdbf4ff9c7a23cac7a2c"
server_key_iv: "059e78dc512f7d446a11fa650f9b37e8"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.395507 7050 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal/instance:
uuid: "2782cbf53f7f4b269547f7f02b84c785"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1bb0868c0c59fdbf4ff9c7a23cac7a2c"
server_key_iv: "059e78dc512f7d446a11fa650f9b37e8"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.396725 7050 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.000s sys 0.001s
I20260923 23:14:15.397375 7065 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:15.397518 7050 fs_manager.cc:725] Time spent opening block manager: real 0.000s user 0.000s sys 0.001s
I20260923 23:14:15.397600 7050 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal
uuid: "2782cbf53f7f4b269547f7f02b84c785"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "1bb0868c0c59fdbf4ff9c7a23cac7a2c"
server_key_iv: "059e78dc512f7d446a11fa650f9b37e8"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.397658 7050 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:15.410916 7043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } attempt: 2
I20260923 23:14:15.411427 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.411270 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58630)
0923 23:14:15.411334 (+ 64us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.411384 (+ 50us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
I20260923 23:14:15.411816 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.411711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58640)
0923 23:14:15.411761 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.411791 (+ 30us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20260923 23:14:15.411908 7043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.189:42301: Network error: Client connection negotiation failed: client connection to 127.16.122.189:42301: connect: Connection refused (error 111)
I20260923 23:14:15.415339 7050 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:15.417719 7050 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:15.417834 7050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:15.421644 7050 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.189:42301
I20260923 23:14:15.421685 7117 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.189:42301 every 8 connection(s)
I20260923 23:14:15.422012 7050 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/data/info.pb
I20260923 23:14:15.422914 7118 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:15.424500 7118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:15.425808 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 7050
I20260923 23:14:15.425909 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-1/wal/instance
I20260923 23:14:15.426105 16874 external_mini_cluster.cc:1428] Setting key 319aaca62673d79565d3ed8816865006
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.122.188@KRBTEST.COM; defaulting to no policy
I20260923 23:14:15.433750 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.425239 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42448 (local address 127.16.122.190:36731)
0923 23:14:15.425300 (+ 61us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.425302 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.425436 (+ 134us) server_negotiation.cc:407] Connection header received
0923 23:14:15.426055 (+ 619us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.426057 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.426072 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.426123 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.426954 (+ 831us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.427502 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.428543 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.428764 (+ 221us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.428830 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.431160 (+ 2330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.431163 (+ 3us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.431163 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.431170 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.432432 (+ 1262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.432831 (+ 399us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.432833 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.432833 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.432886 (+ 53us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.433108 (+ 222us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.433108 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.433109 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.433219 (+ 110us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.433276 (+ 57us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.433569 (+ 293us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.433656 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:14:15.433913 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.425164 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36731 (local address 127.0.0.1:42448)
0923 23:14:15.425321 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.425365 (+ 44us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.425916 (+ 551us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.426216 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.426221 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.426556 (+ 335us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.426838 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.426844 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.427603 (+ 759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.427604 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.428441 (+ 837us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.428444 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.428849 (+ 405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.428850 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.428994 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.429527 (+ 533us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.429537 (+ 10us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.431072 (+ 1535us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.432508 (+ 1436us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.432515 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.432517 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.432763 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.432940 (+ 177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.432941 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.432941 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.433046 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.433315 (+ 269us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.433322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.433451 (+ 129us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.433540 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.433705 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":118,"thread_start_us":68,"threads_started":1}
I20260923 23:14:15.434432 7118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.122.188@KRBTEST.COM" created.
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:15.441459 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.434665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58656)
0923 23:14:15.434736 (+ 71us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.434753 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.434796 (+ 43us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.435082 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.435083 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.435214 (+ 131us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.435268 (+ 54us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.435270 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.436333 (+ 1063us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.436334 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.436833 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.436835 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.437277 (+ 442us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.437278 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.437335 (+ 57us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.437685 (+ 350us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.437690 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.438907 (+ 1217us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.440524 (+ 1617us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.440525 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.440525 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.440672 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.440840 (+ 168us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.440841 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.440841 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.440893 (+ 52us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.441138 (+ 245us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.441139 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.441176 (+ 37us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.441236 (+ 60us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.441331 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20260923 23:14:15.441489 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.434739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58656 (local address 127.16.122.189:42301)
0923 23:14:15.434886 (+ 147us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.434892 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.434914 (+ 22us) server_negotiation.cc:407] Connection header received
0923 23:14:15.434934 (+ 20us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.434936 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.434950 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.435020 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.435326 (+ 306us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.436242 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.436906 (+ 664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.437216 (+ 310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.437271 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.438984 (+ 1713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.438991 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.438992 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.438998 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.440451 (+ 1453us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.440736 (+ 285us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.440739 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.440740 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.440790 (+ 50us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.440939 (+ 149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.440940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.440940 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.441081 (+ 141us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.441151 (+ 70us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.441264 (+ 113us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.441382 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":122,"thread_start_us":82,"threads_started":1}
I20260923 23:14:15.442143 7118 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:15.442476 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.442348 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45612)
0923 23:14:15.442396 (+ 48us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.442447 (+ 51us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
W20260923 23:14:15.442597 7068 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
I20260923 23:14:15.442996 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.442907 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45614)
0923 23:14:15.442948 (+ 41us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.442973 (+ 25us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
W20260923 23:14:15.443099 7118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:39731: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
Entry for principal kudu/127.16.122.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab.
I20260923 23:14:15.449776 16874 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskODLJMS/build/release/bin/kudu
/tmp/dist-test-taskODLJMS/build/release/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:39731
--webserver_interface=127.16.122.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.122.148:45861
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260923 23:14:15.492313 7118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } attempt: 1
I20260923 23:14:15.492794 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.492671 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45620)
0923 23:14:15.492730 (+ 59us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.492764 (+ 34us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20260923 23:14:15.493119 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.493041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45628)
0923 23:14:15.493079 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.493102 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":24}
W20260923 23:14:15.493202 7118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:39731: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
I20260923 23:14:15.503064 7043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } attempt: 3
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:15.510118 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.503390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58664)
0923 23:14:15.503466 (+ 76us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.503479 (+ 13us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.503554 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.503837 (+ 283us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.503838 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.504025 (+ 187us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.504102 (+ 77us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.504104 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.504746 (+ 642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.504747 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.505616 (+ 869us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.505620 (+ 4us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.505982 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.505983 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.506061 (+ 78us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.506502 (+ 441us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.506507 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.507828 (+ 1321us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.509217 (+ 1389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.509218 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.509219 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.509381 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.509547 (+ 166us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.509548 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.509548 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.509605 (+ 57us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.509810 (+ 205us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.509811 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.509860 (+ 49us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.509919 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.510015 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20260923 23:14:15.510118 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.503655 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58664 (local address 127.16.122.189:42301)
0923 23:14:15.503711 (+ 56us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.503713 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.503724 (+ 11us) server_negotiation.cc:407] Connection header received
0923 23:14:15.503750 (+ 26us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.503751 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.503761 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.503810 (+ 49us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.504179 (+ 369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.504673 (+ 494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.505703 (+ 1030us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.505918 (+ 215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.505969 (+ 51us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.507919 (+ 1950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.507920 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.507921 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.507928 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.509138 (+ 1210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.509441 (+ 303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.509442 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.509443 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.509494 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.509653 (+ 159us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.509654 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.509654 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.509758 (+ 104us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.509811 (+ 53us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.509945 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.510015 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:14:15.510643 7043 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } has no permanent_uuid. Determining permanent_uuid...
I20260923 23:14:15.510977 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.510882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45644)
0923 23:14:15.510922 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.510950 (+ 28us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:14:15.511322 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.511245 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45654)
0923 23:14:15.511283 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.511306 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":28}
W20260923 23:14:15.511397 7043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:39731: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
W20260923 23:14:15.531587 7125 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260923 23:14:15.531823 7125 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260923 23:14:15.531847 7125 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260923 23:14:15.531873 7125 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260923 23:14:15.533051 7125 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260923 23:14:15.533089 7125 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260923 23:14:15.533106 7125 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260923 23:14:15.533121 7125 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260923 23:14:15.533136 7125 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260923 23:14:15.534588 7125 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.122.148:45861
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.122.190:36731,127.16.122.189:42301,127.16.122.188:39731
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/krb5kdc/kudu_127.16.122.188.keytab
--principal=kudu/127.16.122.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.122.188:39731
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.122.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--heap_profile_path=/tmp/kudu.7125
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type RELEASE
built by systest at 23 Sep 2026 22:15:01 UTC on ccycloud.quasar-sngwrb.root.comops.site
build id 4668915
I20260923 23:14:15.534833 7125 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260923 23:14:15.535074 7125 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260923 23:14:15.539815 7125 server_base.cc:1034] running on GCE node
W20260923 23:14:15.539846 7131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:15.539938 7130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260923 23:14:15.539875 7133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260923 23:14:15.540280 7125 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260923 23:14:15.540584 7125 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260923 23:14:15.541757 7125 hybrid_clock.cc:648] HybridClock initialized: now 1790205255541738 us; error 34 us; skew 500 ppm
I20260923 23:14:15.542528 7043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } attempt: 1
I20260923 23:14:15.542906 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.542781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45656)
0923 23:14:15.542839 (+ 58us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.542874 (+ 35us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":32}
I20260923 23:14:15.543262 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.543182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45668)
0923 23:14:15.543222 (+ 40us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.543245 (+ 23us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":25}
W20260923 23:14:15.543347 7043 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:39731: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260923 23:14:15.544438 7125 init.cc:377] Logged in from keytab as kudu/127.16.122.188@KRBTEST.COM (short username kudu)
I20260923 23:14:15.544847 7125 webserver.cc:460] Webserver started at http://127.16.122.188:40833/ using document root <none> and password file <none>
I20260923 23:14:15.545022 7125 fs_manager.cc:359] Metadata directory not provided
I20260923 23:14:15.545063 7125 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260923 23:14:15.545146 7125 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260923 23:14:15.546268 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data/instance:
uuid: "aeca1259e1b94e8a81169c12e188be78"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "9f908064ca1582c992501bb142195f1f"
server_key_iv: "c0c1b1676f5727e3f19eefa8574a66ca"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.546571 7125 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal/instance:
uuid: "aeca1259e1b94e8a81169c12e188be78"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "9f908064ca1582c992501bb142195f1f"
server_key_iv: "c0c1b1676f5727e3f19eefa8574a66ca"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.547813 7125 fs_manager.cc:691] Time spent creating directory manager: real 0.001s user 0.003s sys 0.000s
I20260923 23:14:15.548589 7140 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260923 23:14:15.548774 7125 fs_manager.cc:725] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260923 23:14:15.548856 7125 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data,/tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal
uuid: "aeca1259e1b94e8a81169c12e188be78"
format_stamp: "Formatted at 2026-09-23 23:14:15 on dist-test-slave-95hh"
server_key: "9f908064ca1582c992501bb142195f1f"
server_key_iv: "c0c1b1676f5727e3f19eefa8574a66ca"
server_key_version: "encryptionkey@0"
I20260923 23:14:15.548913 7125 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260923 23:14:15.559636 7125 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260923 23:14:15.561332 7118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } attempt: 2
I20260923 23:14:15.561410 7125 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260923 23:14:15.561518 7125 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260923 23:14:15.561705 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.561596 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45674)
0923 23:14:15.561646 (+ 50us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.561675 (+ 29us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":27}
I20260923 23:14:15.562089 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.561954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45678)
0923 23:14:15.562016 (+ 62us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.562057 (+ 41us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20260923 23:14:15.562196 7118 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.122.188:39731: Network error: Client connection negotiation failed: client connection to 127.16.122.188:39731: connect: Connection refused (error 111)
I20260923 23:14:15.565397 7125 rpc_server.cc:307] RPC server started. Bound to: 127.16.122.188:39731
I20260923 23:14:15.565433 7192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.122.188:39731 every 8 connection(s)
I20260923 23:14:15.565761 7125 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/data/info.pb
I20260923 23:14:15.566597 7193 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260923 23:14:15.568169 7193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:15.574390 16874 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskODLJMS/build/release/bin/kudu as pid 7125
I20260923 23:14:15.574504 16874 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskODLJMS/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1790204999454807-16874-0/minicluster-data/master-2/wal/instance
I20260923 23:14:15.574739 16874 external_mini_cluster.cc:1428] Setting key b5baaa4ee03fa8e3b87a319b68337535
I20260923 23:14:15.576855 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.568850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42458 (local address 127.16.122.190:36731)
0923 23:14:15.568934 (+ 84us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.568936 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.569091 (+ 155us) server_negotiation.cc:407] Connection header received
0923 23:14:15.569525 (+ 434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.569526 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.569562 (+ 36us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.569613 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.570184 (+ 571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.570668 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.571658 (+ 990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.571856 (+ 198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.571929 (+ 73us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.573971 (+ 2042us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.573973 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.573973 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.573982 (+ 9us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.575601 (+ 1619us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.575941 (+ 340us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.575942 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.575942 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.576005 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.576230 (+ 225us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.576232 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.576232 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.576353 (+ 121us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.576411 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.576680 (+ 269us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.576763 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20260923 23:14:15.577082 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.568827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.190:36731 (local address 127.0.0.1:42458)
0923 23:14:15.568979 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.569015 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.569445 (+ 430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.569675 (+ 230us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.569678 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.569922 (+ 244us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.570112 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.570115 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.570740 (+ 625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.570740 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.571564 (+ 824us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.571566 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.571928 (+ 362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.571929 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.572023 (+ 94us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.572496 (+ 473us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.572512 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.573878 (+ 1366us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.575666 (+ 1788us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.575668 (+ 2us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.575670 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.575878 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.576067 (+ 189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.576068 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.576068 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.576155 (+ 87us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.576411 (+ 256us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.576418 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.576551 (+ 133us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.576640 (+ 89us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.576864 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":115,"mutex_wait_us":55,"thread_start_us":78,"threads_started":1}
I20260923 23:14:15.577548 7193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:15.583537 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.575150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45688 (local address 127.16.122.188:39731)
0923 23:14:15.575294 (+ 144us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.575298 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.575348 (+ 50us) server_negotiation.cc:407] Connection header received
0923 23:14:15.575425 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.575426 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.575438 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.575494 (+ 56us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.576180 (+ 686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.576870 (+ 690us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.577852 (+ 982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.578158 (+ 306us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.578224 (+ 66us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.580433 (+ 2209us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.580441 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.580442 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.580449 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.581999 (+ 1550us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.582351 (+ 352us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.582354 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.582355 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.582413 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.582600 (+ 187us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.582601 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.582601 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.582752 (+ 151us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.582920 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.583374 (+ 454us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.583452 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116,"thread_start_us":90,"threads_started":1}
I20260923 23:14:15.583843 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.577777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.189:42301 (local address 127.0.0.1:58678)
0923 23:14:15.577815 (+ 38us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.577824 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.577860 (+ 36us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.577997 (+ 137us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.577998 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.578115 (+ 117us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.578163 (+ 48us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.578164 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.578865 (+ 701us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.578866 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.579399 (+ 533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.579401 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.579735 (+ 334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.579736 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.579802 (+ 66us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.580133 (+ 331us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.580137 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.581443 (+ 1306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.582813 (+ 1370us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.582813 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.582814 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.583022 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.583237 (+ 215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.583238 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.583238 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.583302 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.583555 (+ 253us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.583556 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.583596 (+ 40us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.583671 (+ 75us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.583754 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23}
I20260923 23:14:15.583868 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.577863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58678 (local address 127.16.122.189:42301)
0923 23:14:15.577906 (+ 43us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.577908 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.577915 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:14:15.577936 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.577937 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.577946 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.577989 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.578229 (+ 240us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.578799 (+ 570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.579476 (+ 677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.579681 (+ 205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.579744 (+ 63us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.581527 (+ 1783us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.581528 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.581528 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.581534 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.582743 (+ 1209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.583128 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.583129 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.583129 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.583175 (+ 46us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.583375 (+ 200us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.583376 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.583376 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.583490 (+ 114us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.583539 (+ 49us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.583709 (+ 170us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.583781 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:14:15.584388 7193 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } has no permanent_uuid. Determining permanent_uuid...
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.188@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.190@KRBTEST.COM
I20260923 23:14:15.590691 7195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.584634 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45690)
0923 23:14:15.584700 (+ 66us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.584709 (+ 9us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.584743 (+ 34us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.585269 (+ 526us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.585270 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.585391 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.585416 (+ 25us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.585418 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.586011 (+ 593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586012 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.586468 (+ 456us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.586469 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.586799 (+ 330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586799 (+ 0us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.586843 (+ 44us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.587180 (+ 337us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.587184 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.588329 (+ 1145us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.589721 (+ 1392us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.589722 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.589722 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.589878 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.590062 (+ 184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.590062 (+ 0us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.590063 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.590119 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.590373 (+ 254us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.590374 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.590413 (+ 39us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.590466 (+ 53us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.590549 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20260923 23:14:15.590755 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.584664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45690 (local address 127.16.122.188:39731)
0923 23:14:15.585178 (+ 514us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.585179 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.585185 (+ 6us) server_negotiation.cc:407] Connection header received
0923 23:14:15.585199 (+ 14us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.585200 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.585208 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.585255 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.585479 (+ 224us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.585943 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586537 (+ 594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.586738 (+ 201us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586797 (+ 59us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.588401 (+ 1604us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.588402 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.588403 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.588409 (+ 6us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.589648 (+ 1239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.589939 (+ 291us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.589940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.589941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.589992 (+ 51us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.590173 (+ 181us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.590174 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.590174 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.590297 (+ 123us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.590369 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.590505 (+ 136us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.590608 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":21,"server-negotiator.queue_time_us":496}
I20260923 23:14:15.591565 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.584559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42474 (local address 127.16.122.190:36731)
0923 23:14:15.584603 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.584605 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.584614 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:14:15.584635 (+ 21us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.584636 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.584645 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.584688 (+ 43us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.585154 (+ 466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.585707 (+ 553us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586352 (+ 645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.586547 (+ 195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.586599 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.588979 (+ 2380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.588981 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.588981 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.588988 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.590202 (+ 1214us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.590564 (+ 362us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.590566 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.590566 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.590621 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.591000 (+ 379us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.591001 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.591002 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.591180 (+ 178us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.591353 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.591427 (+ 74us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.591496 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24}
I20260923 23:14:15.592411 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: Bootstrap starting.
I20260923 23:14:15.593372 7193 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:15.593677 7193 log.cc:824] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:15.594440 7193 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: No bootstrap required, opened a new log
I20260923 23:14:15.596061 7193 raft_consensus.cc:348] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.596195 7193 raft_consensus.cc:729] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aeca1259e1b94e8a81169c12e188be78, State: Initialized, Role: FOLLOWER
I20260923 23:14:15.596297 7193 consensus_queue.cc:260] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.122.189@KRBTEST.COM
I20260923 23:14:15.596621 7193 sys_catalog.cc:564] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:15.596582 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.596689 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:14:15.598434 7210 catalog_manager.cc:1558] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:15.598489 7210 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:15.599773 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.592266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58686 (local address 127.16.122.189:42301)
0923 23:14:15.592323 (+ 57us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.592325 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.592333 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:14:15.592356 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.592357 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.592367 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.592424 (+ 57us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.592887 (+ 463us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.593566 (+ 679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.594291 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.594574 (+ 283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.594644 (+ 70us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.597054 (+ 2410us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.597055 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.597055 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.597062 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.598552 (+ 1490us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.598959 (+ 407us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.598960 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.598960 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.599023 (+ 63us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.599236 (+ 213us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.599237 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.599237 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.599367 (+ 130us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.599542 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.599579 (+ 37us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.599675 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":34}
I20260923 23:14:15.601086 16874 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260923 23:14:15.598752 7125 master_runner.cc:186] Error getting master registration for 127.16.122.190:36731: OK, instance_id { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" instance_seqno: 1790205255273450 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:14:15.601568 7125 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:14:15.602234 7017 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2f52918c93c94a87a4394306f3c46c8a"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42474
I20260923 23:14:15.618502 7043 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } attempt: 2
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.190@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:14:15.625548 7046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.618836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45706)
0923 23:14:15.618904 (+ 68us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.618916 (+ 12us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.618967 (+ 51us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.619141 (+ 174us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.619142 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.619298 (+ 156us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.619342 (+ 44us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.619343 (+ 1us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.619991 (+ 648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.619992 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.620498 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.620500 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.620815 (+ 315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.620816 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.620871 (+ 55us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.621355 (+ 484us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.621361 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.622874 (+ 1513us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.624585 (+ 1711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.624586 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.624587 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.624749 (+ 162us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.624921 (+ 172us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.624922 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.624923 (+ 1us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.624978 (+ 55us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.625237 (+ 259us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.625238 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.625286 (+ 48us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.625345 (+ 59us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.625445 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20260923 23:14:15.625550 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.618939 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45706 (local address 127.16.122.188:39731)
0923 23:14:15.619010 (+ 71us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.619011 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.619020 (+ 9us) server_negotiation.cc:407] Connection header received
0923 23:14:15.619071 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.619072 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.619082 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.619129 (+ 47us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.619408 (+ 279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.619916 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.620572 (+ 656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.620758 (+ 186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.620813 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.622957 (+ 2144us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.622959 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.622959 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.622967 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.624496 (+ 1529us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.624809 (+ 313us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.624810 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.624811 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.624866 (+ 55us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.625027 (+ 161us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.625027 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.625028 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.625180 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.625239 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.625373 (+ 134us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.625445 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20260923 23:14:15.627084 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: Bootstrap starting.
I20260923 23:14:15.627830 7043 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:15.628115 7043 log.cc:824] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:15.628815 7043 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: No bootstrap required, opened a new log
I20260923 23:14:15.629781 7043 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.629894 7043 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2f52918c93c94a87a4394306f3c46c8a, State: Initialized, Role: FOLLOWER
I20260923 23:14:15.629971 7043 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.630223 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.630275 7043 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:15.630312 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:15.630391 7017 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260923 23:14:15.630479 7017 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:15.631273 7017 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
W20260923 23:14:15.632027 7224 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:15.632133 7224 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:15.632804 7167 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f52918c93c94a87a4394306f3c46c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "aeca1259e1b94e8a81169c12e188be78"
I20260923 23:14:15.632951 7167 raft_consensus.cc:3037] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:15.633847 7167 raft_consensus.cc:2454] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f52918c93c94a87a4394306f3c46c8a in term 1.
I20260923 23:14:15.634262 6993 leader_election.cc:302] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2f52918c93c94a87a4394306f3c46c8a, aeca1259e1b94e8a81169c12e188be78; no voters:
I20260923 23:14:15.634428 7213 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 1 FOLLOWER]: Leader election won for term 1
I20260923 23:14:15.634582 7213 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [term 1 LEADER]: Becoming Leader. State: Replica: 2f52918c93c94a87a4394306f3c46c8a, State: Running, Role: LEADER
I20260923 23:14:15.634673 7213 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.635177 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f52918c93c94a87a4394306f3c46c8a. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.635284 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:15.635628 7017 leader_election.cc:288] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [CANDIDATE]: Term 1 election: Requested vote from peers 2782cbf53f7f4b269547f7f02b84c785 (127.16.122.189:42301), aeca1259e1b94e8a81169c12e188be78 (127.16.122.188:39731)
I20260923 23:14:15.635608 6975 master_runner.cc:186] Error getting master registration for 127.16.122.189:42301: OK, instance_id { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" instance_seqno: 1790205255420555 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260923 23:14:15.637851 6975 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260923 23:14:15.643476 7228 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260923 23:14:15.643711 7228 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260923 23:14:15.645560 7047 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.637956 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42482 (local address 127.16.122.190:36731)
0923 23:14:15.638019 (+ 63us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.638022 (+ 3us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.638030 (+ 8us) server_negotiation.cc:407] Connection header received
0923 23:14:15.638059 (+ 29us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.638060 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.638069 (+ 9us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.638120 (+ 51us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.638920 (+ 800us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.639674 (+ 754us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.640705 (+ 1031us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.640992 (+ 287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.641032 (+ 40us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.642649 (+ 1617us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.642651 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.642652 (+ 1us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.642659 (+ 7us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.644279 (+ 1620us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.644721 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.644723 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.644723 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.644781 (+ 58us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.644971 (+ 190us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.644971 (+ 0us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.644972 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.645119 (+ 147us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.645190 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.645394 (+ 204us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.645471 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20260923 23:14:15.652293 7167 raft_consensus.cc:1261] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [term 1 FOLLOWER]: Refusing update from remote peer 2f52918c93c94a87a4394306f3c46c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:14:15.652645 7226 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:14:15.653314 7118 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } attempt: 3
I20260923 23:14:15.654049 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f52918c93c94a87a4394306f3c46c8a. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.654145 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:15.654794 7122 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.643288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58688 (local address 127.16.122.189:42301)
0923 23:14:15.643336 (+ 48us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.643338 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.646411 (+ 3073us) server_negotiation.cc:407] Connection header received
0923 23:14:15.646509 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.646511 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.646522 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.646549 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.647157 (+ 608us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.647904 (+ 747us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.648900 (+ 996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.649669 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.649713 (+ 44us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.651254 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.651255 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.651255 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.651263 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.652784 (+ 1521us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.653585 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.653587 (+ 2us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.653587 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.653655 (+ 68us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.653939 (+ 284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.653940 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.653941 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.654093 (+ 152us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.654162 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.654605 (+ 443us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.654700 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23}
I20260923 23:14:15.658430 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.636774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45722 (local address 127.16.122.188:39731)
0923 23:14:15.636823 (+ 49us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.636825 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.644687 (+ 7862us) server_negotiation.cc:407] Connection header received
0923 23:14:15.646338 (+ 1651us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.646340 (+ 2us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.646355 (+ 15us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.646414 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.647532 (+ 1118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.648284 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.649768 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.650370 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.650412 (+ 42us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.655072 (+ 4660us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.655074 (+ 2us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.655074 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.655082 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.656790 (+ 1708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.657311 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.657312 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.657312 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.657389 (+ 77us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.657683 (+ 294us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.657684 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.657684 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.657822 (+ 138us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.657918 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.658255 (+ 337us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.658335 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20}
I20260923 23:14:15.658371 7226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.658468 7226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:15.658896 7199 sys_catalog.cc:455] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.658982 7199 sys_catalog.cc:458] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:15.659198 7236 mvcc.cc:205] Tried to move back new op lower bound from 7332680727148023808 to 7332680727080460288. Current Snapshot: MvccSnapshot[applied={T|T < 7332680727148023808}]
I20260923 23:14:15.659432 7237 mvcc.cc:205] Tried to move back new op lower bound from 7332680727148023808 to 7332680727080460288. Current Snapshot: MvccSnapshot[applied={T|T < 7332680727148023808}]
I20260923 23:14:15.659904 7228 catalog_manager.cc:1347] Generated new cluster ID: 758b1185790f46238e27f912a4cb38ff
I20260923 23:14:15.659950 7228 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
Sep 23 23:14:15 dist-test-slave-95hh krb5kdc[6962](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790205255, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.122.189@KRBTEST.COM for kudu/127.16.122.188@KRBTEST.COM
I20260923 23:14:15.667819 7120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.653547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.122.188:39731 (local address 127.0.0.1:45730)
0923 23:14:15.653624 (+ 77us) negotiation.cc:107] Waiting for socket to connect
0923 23:14:15.653634 (+ 10us) client_negotiation.cc:174] Beginning negotiation
0923 23:14:15.653666 (+ 32us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0923 23:14:15.658616 (+ 4950us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0923 23:14:15.658617 (+ 1us) client_negotiation.cc:394] Received NEGOTIATE response from server
0923 23:14:15.658769 (+ 152us) client_negotiation.cc:189] Negotiated authn=SASL
0923 23:14:15.658811 (+ 42us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.658813 (+ 2us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.659655 (+ 842us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.659656 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.660422 (+ 766us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0923 23:14:15.660425 (+ 3us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.660946 (+ 521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.660947 (+ 1us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0923 23:14:15.661018 (+ 71us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.661515 (+ 497us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0923 23:14:15.661520 (+ 5us) client_negotiation.cc:656] calling sasl_client_start()
0923 23:14:15.664824 (+ 3304us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0923 23:14:15.666681 (+ 1857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.666682 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.666682 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.666895 (+ 213us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.667178 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0923 23:14:15.667179 (+ 1us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0923 23:14:15.667179 (+ 0us) client_negotiation.cc:760] calling sasl_client_step()
0923 23:14:15.667248 (+ 69us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0923 23:14:15.667519 (+ 271us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0923 23:14:15.667520 (+ 1us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0923 23:14:15.667566 (+ 46us) client_negotiation.cc:769] Sending connection context
0923 23:14:15.667623 (+ 57us) client_negotiation.cc:240] Negotiation successful
0923 23:14:15.667719 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":61}
I20260923 23:14:15.668100 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:15.653733 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45730 (local address 127.16.122.188:39731)
0923 23:14:15.658508 (+ 4775us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.658510 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.658517 (+ 7us) server_negotiation.cc:407] Connection header received
0923 23:14:15.658535 (+ 18us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.658535 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.658546 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.658596 (+ 50us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.658887 (+ 291us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.659589 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.660483 (+ 894us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:15.660878 (+ 395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:15.660936 (+ 58us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:15.664925 (+ 3989us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0923 23:14:15.664926 (+ 1us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0923 23:14:15.664926 (+ 0us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0923 23:14:15.664934 (+ 8us) server_negotiation.cc:922] calling sasl_server_start()
0923 23:14:15.666610 (+ 1676us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.667004 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.667005 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.667006 (+ 1us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.667091 (+ 85us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0923 23:14:15.667312 (+ 221us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0923 23:14:15.667313 (+ 1us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0923 23:14:15.667313 (+ 0us) server_negotiation.cc:967] calling sasl_server_step()
0923 23:14:15.667440 (+ 127us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0923 23:14:15.667486 (+ 46us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:15.667936 (+ 450us) server_negotiation.cc:299] Negotiation successful
0923 23:14:15.668012 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4760,"thread_start_us":92,"threads_started":1}
I20260923 23:14:15.669544 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: Bootstrap starting.
I20260923 23:14:15.670284 7118 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: Neither blocks nor log segments found. Creating new log.
I20260923 23:14:15.670593 7118 log.cc:824] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: Log is configured to *not* fsync() on all Append() calls
I20260923 23:14:15.676321 7118 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: No bootstrap required, opened a new log
I20260923 23:14:15.678403 7118 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.678536 7118 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2782cbf53f7f4b269547f7f02b84c785, State: Initialized, Role: FOLLOWER
I20260923 23:14:15.678611 7118 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } }
I20260923 23:14:15.678941 7118 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: configured and running, proceeding with master startup.
I20260923 23:14:15.678906 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.678996 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:15.679267 7091 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [term 0 FOLLOWER]: Advancing to term 1
I20260923 23:14:15.680150 7091 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [term 1 FOLLOWER]: Refusing update from remote peer 2f52918c93c94a87a4394306f3c46c8a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260923 23:14:15.680409 7092 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2f52918c93c94a87a4394306f3c46c8a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "2782cbf53f7f4b269547f7f02b84c785"
I20260923 23:14:15.680480 7213 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [LEADER]: Connected to new peer: Peer: permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260923 23:14:15.681162 7092 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2f52918c93c94a87a4394306f3c46c8a in term 1.
I20260923 23:14:15.682304 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2f52918c93c94a87a4394306f3c46c8a. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.682407 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: This master's current role is: FOLLOWER
I20260923 23:14:15.682453 7243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.682503 7243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785 [sys.catalog]: This master's current role is: FOLLOWER
W20260923 23:14:15.683555 7257 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260923 23:14:15.683614 7257 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260923 23:14:15.684854 7228 catalog_manager.cc:1370] Generated new certificate authority record
I20260923 23:14:15.685599 7228 catalog_manager.cc:1504] Loading token signing keys...
I20260923 23:14:15.689486 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2f52918c93c94a87a4394306f3c46c8a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2f52918c93c94a87a4394306f3c46c8a" member_type: VOTER last_known_addr { host: "127.16.122.190" port: 36731 } } peers { permanent_uuid: "2782cbf53f7f4b269547f7f02b84c785" member_type: VOTER last_known_addr { host: "127.16.122.189" port: 42301 } } peers { permanent_uuid: "aeca1259e1b94e8a81169c12e188be78" member_type: VOTER last_known_addr { host: "127.16.122.188" port: 39731 } } }
I20260923 23:14:15.689586 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a [sys.catalog]: This master's current role is: LEADER
I20260923 23:14:15.690595 7244 mvcc.cc:205] Tried to move back new op lower bound from 7332680727257460736 to 7332680727080460288. Current Snapshot: MvccSnapshot[applied={T|T < 7332680727148023808}]
I20260923 23:14:15.691133 7228 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2f52918c93c94a87a4394306f3c46c8a: Generated new TSK 0
I20260923 23:14:15.691316 7228 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260923 23:14:15.718765 7197 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.717066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45740 (local address 127.16.122.188:39731)
0923 23:14:15.717113 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.717115 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.718085 (+ 970us) server_negotiation.cc:407] Connection header received
0923 23:14:15.718190 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.718191 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.718202 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.718232 (+ 30us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.718690 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:45740: BlockingRecv error: recv got EOF from 127.0.0.1:45740 (error 108)
Metrics: {"server-negotiator.queue_time_us":21}
W20260923 23:14:15.719507 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.717479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.16.122.189:42301)
0923 23:14:15.717518 (+ 39us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.717520 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.719003 (+ 1483us) server_negotiation.cc:407] Connection header received
0923 23:14:15.719122 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.719123 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.719134 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.719161 (+ 27us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.719445 (+ 284us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58704: BlockingRecv error: recv got EOF from 127.0.0.1:58704 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:14:15.720175 7047 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.717824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42486 (local address 127.16.122.190:36731)
0923 23:14:15.717862 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.717864 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.719722 (+ 1858us) server_negotiation.cc:407] Connection header received
0923 23:14:15.719807 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.719808 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.719819 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.719845 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.720114 (+ 269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42486: BlockingRecv error: recv got EOF from 127.0.0.1:42486 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:14:15.722404 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.721330 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58716 (local address 127.16.122.189:42301)
0923 23:14:15.721361 (+ 31us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.721363 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.721916 (+ 553us) server_negotiation.cc:407] Connection header received
0923 23:14:15.722010 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.722011 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.722028 (+ 17us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.722054 (+ 26us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.722356 (+ 302us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58716: BlockingRecv error: recv got EOF from 127.0.0.1:58716 (error 108)
Metrics: {"server-negotiator.queue_time_us":14}
W20260923 23:14:15.725239 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.724527 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58720 (local address 127.16.122.189:42301)
0923 23:14:15.724571 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.724572 (+ 1us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.724792 (+ 220us) server_negotiation.cc:407] Connection header received
0923 23:14:15.724877 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.724878 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.724886 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.724911 (+ 25us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.725192 (+ 281us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58720: BlockingRecv error: recv got EOF from 127.0.0.1:58720 (error 108)
Metrics: {"server-negotiator.queue_time_us":29}
W20260923 23:14:15.729156 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.728342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58722 (local address 127.16.122.189:42301)
0923 23:14:15.728375 (+ 33us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.728377 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.728638 (+ 261us) server_negotiation.cc:407] Connection header received
0923 23:14:15.728736 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.728737 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.728748 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.728776 (+ 28us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.729105 (+ 329us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58722: BlockingRecv error: recv got EOF from 127.0.0.1:58722 (error 108)
Metrics: {"server-negotiator.queue_time_us":16}
W20260923 23:14:15.735276 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.734371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58726 (local address 127.16.122.189:42301)
0923 23:14:15.734409 (+ 38us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.734411 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.734681 (+ 270us) server_negotiation.cc:407] Connection header received
0923 23:14:15.734779 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.734780 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.734793 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.734822 (+ 29us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.735218 (+ 396us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58726: BlockingRecv error: recv got EOF from 127.0.0.1:58726 (error 108)
Metrics: {"server-negotiator.queue_time_us":19}
W20260923 23:14:15.745739 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.744697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58730 (local address 127.16.122.189:42301)
0923 23:14:15.744743 (+ 46us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.744745 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.745039 (+ 294us) server_negotiation.cc:407] Connection header received
0923 23:14:15.745193 (+ 154us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.745194 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.745207 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.745238 (+ 31us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.745663 (+ 425us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58730: BlockingRecv error: recv got EOF from 127.0.0.1:58730 (error 108)
Metrics: {"server-negotiator.queue_time_us":22}
W20260923 23:14:15.764132 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.763178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58732 (local address 127.16.122.189:42301)
0923 23:14:15.763222 (+ 44us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.763224 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.763504 (+ 280us) server_negotiation.cc:407] Connection header received
0923 23:14:15.763633 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.763634 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.763648 (+ 14us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.763680 (+ 32us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.764070 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58732: BlockingRecv error: recv got EOF from 127.0.0.1:58732 (error 108)
Metrics: {"server-negotiator.queue_time_us":20}
W20260923 23:14:15.798318 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.797481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58734 (local address 127.16.122.189:42301)
0923 23:14:15.797546 (+ 65us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.797548 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.797734 (+ 186us) server_negotiation.cc:407] Connection header received
0923 23:14:15.797796 (+ 62us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.797797 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.797810 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.797874 (+ 64us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.798264 (+ 390us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58734: BlockingRecv error: recv got EOF from 127.0.0.1:58734 (error 108)
Metrics: {"server-negotiator.queue_time_us":23}
W20260923 23:14:15.864182 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.863509 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58746 (local address 127.16.122.189:42301)
0923 23:14:15.863588 (+ 79us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.863590 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.863635 (+ 45us) server_negotiation.cc:407] Connection header received
0923 23:14:15.863686 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.863686 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.863696 (+ 10us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.863766 (+ 70us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.864118 (+ 352us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58746: BlockingRecv error: recv got EOF from 127.0.0.1:58746 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
W20260923 23:14:15.993803 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:15.993160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.16.122.189:42301)
0923 23:14:15.993218 (+ 58us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:15.993220 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:15.993297 (+ 77us) server_negotiation.cc:407] Connection header received
0923 23:14:15.993350 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:15.993351 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:15.993362 (+ 11us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:15.993431 (+ 69us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:15.993754 (+ 323us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58762: BlockingRecv error: recv got EOF from 127.0.0.1:58762 (error 108)
Metrics: {"server-negotiator.queue_time_us":30}
W20260923 23:14:16.251631 7122 negotiation.cc:337] Failed RPC negotiation. Trace:
0923 23:14:16.250940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58774 (local address 127.16.122.189:42301)
0923 23:14:16.251023 (+ 83us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.251025 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.251056 (+ 31us) server_negotiation.cc:407] Connection header received
0923 23:14:16.251124 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.251124 (+ 0us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.251136 (+ 12us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.251182 (+ 46us) server_negotiation.cc:226] Negotiated authn=SASL
0923 23:14:16.251570 (+ 388us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58774: BlockingRecv error: recv got EOF from 127.0.0.1:58774 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
I20260923 23:14:16.598896 7210 catalog_manager.cc:1259] Loaded cluster ID: 758b1185790f46238e27f912a4cb38ff
I20260923 23:14:16.598973 7210 catalog_manager.cc:1552] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: loading cluster ID for follower catalog manager: success
I20260923 23:14:16.599835 7210 catalog_manager.cc:1574] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: acquiring CA information for follower catalog manager: success
I20260923 23:14:16.600124 7210 catalog_manager.cc:1602] T 00000000000000000000000000000000 P aeca1259e1b94e8a81169c12e188be78: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:14:16.684036 7257 catalog_manager.cc:1259] Loaded cluster ID: 758b1185790f46238e27f912a4cb38ff
I20260923 23:14:16.684113 7257 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: loading cluster ID for follower catalog manager: success
I20260923 23:14:16.684907 7257 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: acquiring CA information for follower catalog manager: success
I20260923 23:14:16.685209 7257 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 2782cbf53f7f4b269547f7f02b84c785: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260923 23:14:16.767324 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:16.764806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58782 (local address 127.16.122.189:42301)
0923 23:14:16.764932 (+ 126us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.764934 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.764944 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:14:16.765014 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.765015 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.765072 (+ 57us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.765133 (+ 61us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:16.765286 (+ 153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.765787 (+ 501us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.766545 (+ 758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.766751 (+ 206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.766805 (+ 54us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:16.766968 (+ 163us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:16.767098 (+ 130us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:16.767167 (+ 69us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:16.767214 (+ 47us) server_negotiation.cc:299] Negotiation successful
0923 23:14:16.767236 (+ 22us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107,"thread_start_us":62,"threads_started":1}
I20260923 23:14:16.771212 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:16.768885 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45744 (local address 127.16.122.188:39731)
0923 23:14:16.768993 (+ 108us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.768995 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.769005 (+ 10us) server_negotiation.cc:407] Connection header received
0923 23:14:16.769046 (+ 41us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.769047 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.769091 (+ 44us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.769160 (+ 69us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:16.769268 (+ 108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.769757 (+ 489us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.770454 (+ 697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.770668 (+ 214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.770720 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:16.770859 (+ 139us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:16.770971 (+ 112us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:16.771059 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:16.771105 (+ 46us) server_negotiation.cc:299] Negotiation successful
0923 23:14:16.771126 (+ 21us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87,"thread_start_us":48,"threads_started":1}
I20260923 23:14:16.775321 7326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:16.772773 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58786 (local address 127.16.122.189:42301)
0923 23:14:16.772820 (+ 47us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.772822 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.772897 (+ 75us) server_negotiation.cc:407] Connection header received
0923 23:14:16.773009 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.773010 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.773023 (+ 13us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.773075 (+ 52us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:16.773182 (+ 107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.773696 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.774565 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.774857 (+ 292us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.774912 (+ 55us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:16.775071 (+ 159us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:16.775105 (+ 34us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:16.775150 (+ 45us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:16.775218 (+ 68us) server_negotiation.cc:299] Negotiation successful
0923 23:14:16.775244 (+ 26us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22}
I20260923 23:14:16.775806 7332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:16.772743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45756 (local address 127.16.122.188:39731)
0923 23:14:16.772888 (+ 145us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.772890 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.773151 (+ 261us) server_negotiation.cc:407] Connection header received
0923 23:14:16.773174 (+ 23us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.773175 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.773183 (+ 8us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.773225 (+ 42us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:16.773382 (+ 157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.774065 (+ 683us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.774897 (+ 832us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.775208 (+ 311us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.775260 (+ 52us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:16.775421 (+ 161us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:16.775450 (+ 29us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:16.775573 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:16.775725 (+ 152us) server_negotiation.cc:299] Negotiation successful
0923 23:14:16.775748 (+ 23us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20260923 23:14:16.775794 7338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0923 23:14:16.772687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42492 (local address 127.16.122.190:36731)
0923 23:14:16.772806 (+ 119us) server_negotiation.cc:206] Beginning negotiation
0923 23:14:16.772808 (+ 2us) server_negotiation.cc:399] Waiting for connection header
0923 23:14:16.772857 (+ 49us) server_negotiation.cc:407] Connection header received
0923 23:14:16.772989 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0923 23:14:16.772990 (+ 1us) server_negotiation.cc:461] Received NEGOTIATE request from client
0923 23:14:16.773042 (+ 52us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0923 23:14:16.773119 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0923 23:14:16.773228 (+ 109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.773988 (+ 760us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.774834 (+ 846us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0923 23:14:16.775155 (+ 321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0923 23:14:16.775216 (+ 61us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0923 23:14:16.775382 (+ 166us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0923 23:14:16.775492 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0923 23:14:16.775553 (+ 61us) server_negotiation.cc:1035] Waiting for connection context
0923 23:14:16.775603 (+ 50us) server_negotiation.cc:299] Negotiation successful
0923 23:14:16.775627 (+ 24us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97,"thread_start_us":42,"threads_started":1}
I20260923 23:14:16.781530 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 6975
I20260923 23:14:16.786991 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 7050
I20260923 23:14:16.791409 16874 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskODLJMS/build/release/bin/kudu with pid 7125
2026-09-23T23:14:16Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (2695 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (5436 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (257343 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST